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Abstract 


Due  to  the  large  numbers  (up  to  tens  of  thousands)  of  possible  targets  identified  in  nominal  UXO 
surveys,  efficient  and  reliable  machine-aided  target  pickers  should  be  used  to  identify  targets  for 
subsequent  characterization.  When  selecting  anomalies,  the  goal  is  to  identify  all  anomalous 
features  that  may  be  caused  by  UXO  while  minimizing  operator  time  and  eliminating  operator 
bias.  To  facilitate  advanced  physics-based  modeling,  however,  the  target  pickers  should  also  be 
able  to  select  data  appropriate  to  the  target,  i.e.,  to  outline  or  estimate  the  anomalies  spatial 
extent.  The  current  approach  to  target  selection  is  either  manual  identification  or  amplitude 
thresholding.  The  former  is  time  intensive,  not  clearly  defined,  and  prone  to  operator  bias.  The 
latter  is  sensitive  to  noise  and  is  prone  to  over-  or  under-picking  unless  judicious  oversight  is 
exercised.  Neither  approach  provides  measures  for  estimating  the  footprint  of  the  anomaly.  The 
impact  to  the  DoD  is  obvious.  Systematic,  fast,  and  robust  target  pickers  can  save  money  and 
produce  a  defensible  target  list  compared  to  the  current  methods. 

This  project  evaluated  four  automatic  target  pickers  against  the  manual  method  and  transitioned 
them  to  the  user  community  via  Oasis  montaj™  by  building  custom  Geosoft  Executables  (GX). 
Oasis  montaj™  is  a  geophysical  data  processing  and  visualization  package  developed  and 
marketed  by  Geosoft  Incorporated.  The  four  automatic  target  pickers  were:  (1)  a  wavelet-based 
detection  algorithm,  (2)  clustering  positive  and  negative  peaks,  (3)  a  dipole-based  matched  fdter, 
and  (4)  analytic  signal. 

The  demonstration  was  broken  up  into  two  phases.  The  first  phase  used  a  60  dipole  synthetic 
data  set  to  explore  the  parameter  space  and  optimize  the  algorithms  for  the  four  automatic  target 
pickers.  The  result  of  phase  one  was  a  set  of  starting  parameters  that  was  used  in  phase  two. 

The  second  phase  applied  the  target  pickers  to  seven  magnetic  data  sets  using  the  parameters 
output  from  phase  one  as  a  starting  point.  The  seven  data  sets  possessed  different  signal  and 
noise  characteristics  and  anomaly  densities.  Three  datasets  provided  from  the  Corps  of 
Engineers  (COE),  a  helicopter-towed  Wide  Area  Assessment  (WAA)  dataset,  a  vehicle-towed 
transect  WAA  dataset,  and  vehicle-towed  Multi-sensor  Towed  Array  System  (MTADS)  datasets 
from  the  Aberdeen  Proving  Ground  (APG)  standardized  test  site  and  Target  SI  at  fsleta  Pueblo 
in  New  Mexico  were  used  for  the  evaluation.  Because  each  data  set  has  its  own  unique  data 
characteristics  the  starting  parameters  were  adjusted  iteratively  to  achieve  the  best  performance. 
The  knowledge  gained  from  phase  one  was  used  to  guide  these  adjustments. 

Overall,  the  manual  method  proved  to  be  the  best  at  picking  valid  targets  especially  in  areas  with 
varying  amounts  of  geology  and  background  noise.  In  general,  the  manual  picker  was  able  to  set 
a  lower  threshold  for  each  data  set  than  the  automatic  methods  because  he  can  screen  out  the 
anomalies  with  signal  amplitudes  above  a  threshold  that  are  caused  by  noisy  data  or  geology  and 
pick  the  small  amplitude  anomalies  located  in  the  quieter  areas.  This  ability  is  also  useful  when 
dealing  with  poor  quality  data.  The  manual  picker  was  able  to  interpret  positioning  problems 
that  caused  some  large  anomalies  to  appear  to  be  several  small  anomalies  and  only  make  one 
declaration  whereas  some  of  the  automatic  pickers  had  multiple  declarations.  The  main 
drawbacks  to  the  manual  method  are  time,  operator  bias  and  operator  error. 
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Of  the  automatic  methods  the  analytic  signal  and  wavelet  method  gave  the  best  results  overall 
but  each  of  the  methods  had  their  own  strengths  and  weaknesses  and  the  best  method  to  use  was 
very  data  dependent.  A  general  observation  for  all  the  automatic  pickers  is  that  they  should  not 
be  run  blindly.  The  analyst  should  carefully  choose  their  parameters  and  analyze  the  results. 

The  process  of  iteratively  changing  parameters  and  visual  review  of  the  results  was  essential  in 
selecting  the  best  parameters. 

The  wavelet  method  worked  well  on  a  range  of  target  sizes  and  densities.  It  performed  better 
than  the  other  methods  at  picking  targets  that  were  clustered.  It  was  fairly  quick  to  run  on  total 
coverage  surveys  but  much  less  efficient  when  run  on  transect  data.  Its  main  weaknesses  were 
the  complexity  of  the  parameters  and  over  picking  in  areas  with  geology,  noise  or  poor  data 
quality. 

The  clustering  method  performed  well  detecting  isolated  targets  with  similar  sizes.  When  a 
range  of  target  sizes  were  encountered  and  the  parameters  were  set  to  detect  the  large  anomalies 
any  small  anomaly  close  to  the  large  anomaly  tended  to  be  missed.  The  clustering  algorithm  also 
had  problems  picking  anomalies  in  areas  with  high  geology  and  noise.  Overall,  it  ran  slower 
than  the  analytic  signal  on  total  survey  areas  but  was  more  efficient  on  transect  data  because  it 
works  on  the  profde  data  and  not  a  grid.  It  also  took  longer  to  setup  because  there  were  multiple 
parameters  that  need  to  be  tuned. 

The  matched  filter  method  performed  the  worst.  It  was  hindered  by  its  requirement  to  have  total 
data  coverage  within  the  filter  box.  This  would  increase  geophysical  data  collection  costs 
because  additional  data  would  need  to  be  collected  around  the  perimeter  of  the  survey  area.  It 
ran  the  slowest  of  the  automatic  methods  but  the  limited  number  of  parameters  helped  reduce  its 
complexity.  It  had  difficulty  with  overlapping  targets,  range  of  target  sizes  and  poor  data  quality. 
The  one  area  it  excelled  at  was  picking  targets  in  areas  with  a  variety  of  geology. 

Implementation  of  the  automatic  picking  methods  used  in  this  demonstration  should 
considerably  reduce  the  time  and  thus  cost  required  to  pick  anomalies  when  compared  to  the 
manual  method.  The  amount  of  cost  savings  will  depend  on  the  data.  In  areas  with  isolated 
anomalies  and  low  background  noise  or  geology  the  cost  savings  will  be  maximized  because  the 
automatic  pickers  are  able  to  detect  over  90%  of  the  anomalies  in  a  fraction  of  the  time  compared 
to  the  manual  method.  As  the  geologic  noise  increases  or  data  quality  decreases  the  cost  savings 
will  diminish  but  still  should  be  significant. 

In  conclusion,  the  best  use  of  the  automatic  target  pickers  may  be  to  quickly  and  consistently 
select  all  the  strong  isolated  targets  by  setting  the  picking  parameters  to  minimize  false  alarms. 
The  remaining  anomalies  can  then  be  selected  using  the  manual  method.  This  combines  the 
strengths  of  each  method  and  will  result  in  a  better  product  than  using  only  the  automatic 
methods  and  cost  savings  compared  to  using  only  the  manual  method. 
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MM-0502  Final  Report 
Target  Picking  Methods  for  Magnetic  Data 

1.  Introduction 


1.1  Background 

Buried  Unexploded  Ordnance  (UXO)  is  arguably  one  of  the  most  serious  and  prevalent 
environmental  problems  currently  facing  Department  of  Defense  (DoD)  facility  managers.  Not 
limited  to  active  sites  and  test  ranges,  these  problems  also  occur  at  DoD  sites  that  are  currently 
inactive  and  in  areas  adjacent  to  military  ranges  that  belong  to  the  civilian  sector  or  are  under 
control  of  other  government  agencies.  The  exact  amount  of  land  affected  is  uncertain,  but  it  is 
generally  agreed  to  be  in  excess  of  10  million  acres  in  the  continental  United  States.  UXO 
mitigation  and  remediation  requirements  assume  even  more  compelling  proportions  when  the 
DoD  lands  involve  Fonnerly  Used  Defense  Sites  (FUDS)  or  Base  Realignment  and  Closure 
(BRAC)  sites.  These  sites  must  be  certified  as  suitable  for  the  end  use  depending  on  the  pending 
disposition.  Oversight  and  evaluation  of  these  processes  involve  non-DoD  agencies  including 
the  EPA;  state,  county,  and  local  governments;  and  the  civilian  community. 

Due  to  the  large  numbers  (up  to  tens  of  thousands)  of  possible  targets  identified  in  nominal  UXO 
surveys,  efficient  and  reliable  machine-aided  target  pickers  should  be  used  to  identify  targets  for 
subsequent  characterization.  When  selecting  anomalies,  the  goal  is  to  identify  all  anomalous 
features  that  may  be  caused  by  UXO  while  minimizing  operator  time  and  eliminating  operator 
bias.  To  facilitate  advanced  physics-based  modeling,  however,  the  target  pickers  should  also  be 
able  to  select  data  appropriate  to  the  target,  i.e.,  to  outline  or  estimate  the  anomalies  spatial 
extent. 

The  current  approach  to  target  selection  is  either  manual  identification  or  amplitude  thresholding. 
The  former  is  time  intensive,  not  clearly  defined,  and  prone  to  operator  bias.  The  latter  is 
sensitive  to  noise  and  is  prone  to  over-  or  under-picking  unless  judicious  oversight  is  exercised. 
Neither  approach  provides  measures  for  estimating  the  footprint  of  the  anomaly.  The  impact  to 
the  DoD  is  obvious.  Systematic,  fast,  and  robust  target  pickers  save  money  and  produce  a 
defensible  target  list  -  the  alternatives  do  not. 

This  project  evaluated  four  automatic  target  pickers  and  transitioned  them  to  the  user  community 
via  Oasis  montaj™  by  building  custom  Geosoft  Executables  (GX).  Oasis  montaj™  is  a 
geophysical  data  processing  and  visualization  package  developed  and  marketed  by  Geosoft 
Incorporated.  It  has  a  large  capacity  database,  a  professional  graphic  interface,  and  an 
established  client  base.  We  decided  to  transition  research  algorithms  via  Oasis  montaj™  in  order 
to  leverage  its  significant  capabilities,  marketing  channels,  and  customer  support  services. 
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1.2  Objectives  of  the  Demonstration 


This  ESTCP  project  is  aimed  at  evaluating  four  automatic  target-picking  algorithms  for  total 
field  magnetometer  data  and  comparing  the  detection  results  with  ground  truth  when  available  or 
to  anomalies  chosen  manually  by  a  commercial  expert.  The  four  picking  methods  included  in 
the  study  are  (1)  a  wavelet-based  detection  algorithm,  (2)  a  rule-based,  clustering  approach,  (3)  a 
dipole-based  matched  filter,  and  (4)  a  thresholded  analytic  signal  algorithm.  Our  goal  is  to 
identify  all  anomalous  features  that  may  be  caused  by  UXO  while  minimizing  operator  time  and 
eliminating  operator  bias.  Simply  identifying  geophysical  anomalies  is  not  enough,  however  to 
facilitate  advanced  physics-based  modeling,  the  target  picking  routines  should  also  be  able  to 
identify  the  anomalies  spatial  extent. 

The  project  ran  each  target  picking  routine  on  demonstration  data  sets  that  possessed  different 
signal  and  noise  characteristics  and  anomaly  densities.  Three  datasets  provided  from  the  Corps 
of  Engineers  (COE),  a  helicopter-towed  Wide  Area  Assessment  (WAA)  dataset,  a  vehicle-towed 
transect  WAA  dataset,  and  vehicle-towed  Multi-sensor  Towed  Array  System  (MTADS)  datasets 
from  the  Aberdeen  Proving  Ground  (APG)  standardized  test  site  and  Target  SI  at  Isleta  Pueblo 
in  New  Mexico  were  used  for  the  evaluation. 


1.3  Regulatory  Drivers 

The  Senate  Report  (Report  106-50),  pages  291-293,  accompanying  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2000  (Public  Law  106-65),  included  a  provision  entitled 
“Research  and  development  to  support  unexploded  ordnance  clearance,  active  range  unexploded 
ordnance  clearance,  and  explosive  ordnance  disposal.”  This  provision  requires  the  Secretary  of 
Defense  to  submit  to  the  Congressional  Defense  Committees  a  report  that  gives  a  complete 
estimate  of  the  current  and  projected  costs,  including  funding  shortfalls,  for  UXO  response  at 
active  facilities  and  installations  subject  to  BRAC,  and  FUDS. 

The  following  statements  are  taken  verbatim  out  of  the  DoDs  2001  Report  to  Congress: 

“Decades  of  military  training,  exercises,  and  testing  of  weapons  systems  has  required 
that  we  begin  to  focus  our  response  on  the  challenges  of  UXO.  Land  acreage 
potentially  containing  UXO  has  grown  to  include  active  military  sites  and  land 
transferring  or  transferred  for  private  use,  such  as  Base  Realignment  and  Closure 
(BRAC)  sites  and  Formerly  Used  Defense  Sites  (FUDS).  DoD  responsibilities  include 
protecting  personnel  and  the  public  from  explosive  safety  hazards;  UXO  site  cleanup 
project  management;  ensuring  compliance  with  federal,  state,  and  local  laws  and 
environmental  regulations;  assumption  of  liability;  and  appropriate  interactions  with 
the  public. 

...Through  limited  experience  gained  in  executing  these  activities,  it  has  become 
increasingly  clear  that  the  full  size  and  extent  of  the  impact  of  sites  containing  UXO 
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is  yet  to  be  realized.  ...  DoD  has  completed  an  initial  baseline  estimate  for  UXO 
remediation  cost.  This  report  provides  a  UXO  response  estimate  in  a  range  between 
$106.9  billion  and  $391  billion  in  current  year  [2001]  dollars.  ...Technology 
discovery,  development,  and  commercialization  offers  some  hope  that  the  cost  range 
can  be  decreased.  ... 

...  Objective:  Develop  standards  and  protocols  for  navigation,  geo-location,  data 
acquisition  and  processing,  and  performance  of  UXO  technologies. 

•  Standard,  high  quality  archived  data  are  needed  for  optimal  data  processing  of 
geophysical  data,  re-acquisition  for  response  activities,  quality  assurance,  quality 
control,  and  review  by  all  stakeholders.  In  addition  standards  and  protocols  are 
required  for  evaluating  UXO  technology  performance  to  aid  in  selecting  the  most 
effective  technologies  for  individual  sites. 

•  Standard  software  and  visualization  tools  are  needed  to  provide  regulatory  and 
public  visibility  to  and  understanding  of  the  analysis  and  decision  process  made  in 
response  activities.  ” 


1.4  Stakeholder/End-User  Issues 

The  stakeholders  and  end-users  of  this  data  processing  and  analysis  technology  include  private 
contractors  that  conduct  geophysical  investigations  in  support  of  UXO  clean  up  programs  and 
governmental  employees  that  provide  technical  oversight.  This  demonstration  will  introduce  the 
stakeholders  and  end-users  to  data  products  associated  with  this  analysis  approach,  and  to  the 
inherent  transparency  of  the  target  picking  process.  This  basic  information  will  help  to  improve 
the  results  of  future  geophysical  investigations  conducted  by  others.  The  market  for  this  type  of 
guidance  document  includes  all  practicing  geophysical  service  firms  currently  working  in  the 
UXO  industry. 
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2.  Technology  Description 


2.1  Technology  Developments  and  Application 

The  user  community  would  benefit  from  an  algorithm  that  accurately  identifies  and  locates 
magnetic  anomalies  using  quantifiable  decision  criteria  and  limited  user  interactions.  The 
underlying  aim  is  to  reduce  analysts’  bias  because  it  is  not  predictable,  consistent,  or  correctable. 
This  project  made  and  compared  anomaly  declarations  for  each  dataset  chosen  using  the 
following  five  approaches:  (1)  a  wavelet-based  detection  algorithm,  (2)  clustering  positive  and 
negative  peaks,  (3)  a  dipole-based  matched  filter,  (4)  analytic  signal,  and  (5)  manual  selection. 
Each  approach  is  described  below.  We  will  also  evaluate  how  each  method  can  be  used  to 
identify  the  anomalies  spatial  extent. 


Wavelet-based  Algorithm:  Billings  and  Hernnann  (2003)  developed  a  magnetic  target  picking 
method  called  the  Automated  Wavelet  Detection  (AWD)  algorithm.  In  that  method,  individual 
peaks  in  the  magnetic  data  are  followed  across  multiple  scales,  with  the  decay  in  peak  amplitude 
related  to  the  depth  to  the  source.  Nearby  positive  and  negative  peaks  in  the  image  are  only 
joined  together  if  they  satisfy  two  conditions:  (i)  the  peaks  must  have  comparable  depth 
estimates  and  (ii)  the  peaks  must  move  towards  each  other  as  the  wavelet  scale  becomes  finer.  In 
this  way,  one  can  avoid  incorrectly  joining  the  peaks  from  nearby  dipoles.  In  the  last  stage  of  the 
algorithm,  the  amplitudes  of  the  peaks  and  their  relative  position  are  used  to  provide  an  initial 
estimate  of  the  dipole  parameters  including  a  good  initial  estimate  of  the  size  of  the  region  to 
invert  about  each  anomaly. 

The  AWD  method  has  a  solid  theoretical  basis  by  using  the  Poisson  wavelet  (Moreau,  et  ah, 
1999).  This  wavelet  has  been  extensively  studied  and  applied  to  interpretation  of  potential  fields 
(see  Moreau,  et  ah,  1999  and  references  therein),  particularly  for  locating  and  characterizing 
edges  in  the  data  (Hornby  et  al.,  1999).  A  very  useful  property  of  this  class  of  wavelets  is  that 
the  wavelet  transfonn  can  be  simply  obtained  by  upward  continuation  followed  by  horizontal 
differentiation.  This  means  that  the  wavelet  coefficients  at  multiple  scales  can  be  easily  and 
rapidly  computed. 

The  algorithm  consists  of  the  following  five  steps: 

1 .  Follow  local  maxima  and  minima  as  the  image  is  zoomed  to  different  scales  with  a 
wavelet  transform; 

2.  Estimate  the  depth  to  the  source  of  each  peak  by  analyzing  how  the  amplitude  of  the  peak 
changes  with  scale; 

3.  Join  nearby  positive  and  negative  peaks  of  comparable  depth;  and 

4.  Else  the  amplitudes  and  distance  between  adjacent  peaks  to  provide  an  initial  estimate  of 
dipole  parameters. 

5.  Using  the  initial  estimate  of  dipole  parameters,  delineate  the  region  that  comprises  97.5% 
of  the  dipole  energy,  and  resolve  any  issues  of  overlapping  anomalies. 
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Rule-based  Clusters:  Rule-based  clustering  approaches  threshold  the  magnetic  data  to  identify 
positive  and  negative  groups  and  then  associate  positive  and  negative  clusters  using  codified 
rules  to  form  anomalies.  The  degree  to  which  they  work  depends  on  the  noise  and  signal 
characteristics  of  the  data  and  on  details  of  the  rules  that  associate  negative  and  positive  clusters. 
We  have  identified  two  rule-based  schemes  that  have  been  developed  and  codified  under 
previous  research  efforts. 

The  first  is  an  automatic  target  picker  developed  during  SERDP  CU-1092  by  Blackhawk 
Geometries,  Inc.  In  this  version,  thresholds  for  positive  and  negative  data  are  set  by  the  user. 

The  negative  and  positive  clusters  are  then  identified,  and  each  negative  anomaly  is  associated 
with  the  most  plausible  nearby  positive  anomaly.  The  most  plausible  positive  anomaly  is 
determined  by  a  formula  that  reflects  the  distance,  magnitude  of  both  anomalies,  and  degree  of 
consistency  with  the  local  magnetic  declination.  Isolated  negative  anomalies  are  discarded  as 
noise.  As  detailed  in  the  CU-1092  project  report,  there  are  issues  with  regard  to  matching 
positive  and  negative  lobes  in  cases  where  the  anomalies  are  tightly  packed  and/or  overlapping. 
There  are  also  issues  with  regard  to  the  picker  dividing  one  anomaly  into  two  if  there  are  minor 
amplitude  sags  (due  perhaps  to  inter-sensor  difference,  heading  errors,  or  gridding). 

Additionally,  the  Blackhawk  formula-based  codes  were  written  for  the  Silicon  Graphics 
Incorporated  (SGI)  IRIX  operating  system  or  the  LINUX  operating  system. 

The  second  rule-based  scheme  was  developed  and  codified  by  AETC  Incorporated  under  funding 
from  Naval  Explosive  Ordnance  Disposal  Technology  Division  (NAVEODTECHDIV)  in  1995. 
Similar  to  the  code  developed  under  CU-1092,  it  asks  the  user  to  identify  thresholds,  uses  the 
thresholds  to  create  positive  and  negative  clusters,  and  then  associates  the  clusters  in  favor  of 
combinations  with  the  least  distance  between  centers. 

Dipole  based  Matched  Filter:  AETC  developed,  under  ESTCP  Project  199918,  a  Matched  Filter 
AutoProcessor  (MFAP)  for  magnetometer  data.  This  algorithm  implements  a  matched  filter 
based  on  a  magnetic  dipole  signal.  After  the  magnetometer  data  is  interpolated  onto  a  regular  x- 
y  grid,  it  is  automatically  convolved  with  a  magnetic  dipole  signal  model  for  a  target 
accompanied  by  a  search  over  the  unknown  parameters  in  the  signal  model  (target  orientation, 
moment,  etc.)  to  maximize  the  filter  output.  The  output  of  the  matched  filter  routine  is  a  surface 
(grid)  that  peaks  directly  over  the  target  location.  Because  of  this,  it  can  be  thresholded  to 
identify  targets.  The  Matched  Filter  approach  was  developed  in  the  Interactive  Data  Language 
(IDL;  Research  Systems,  Inc.)  but  has  subsequently  been  integrated  with  OASIS  montaj™. 

Once  the  matched  filter  output  is  calculated,  the  user  must  identify  individual  anomalies  by 
picking  peaks  above  the  background  noise.  The  spatial  extent  of  each  anomaly  can  be  estimated 
by  using  the  initial  fitted  parameters  output  during  the  match  fitting  process  to  calculate  a 
boundary  that  contains  a  certain  percentage  of  the  total  energy. 
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Analytic  Signal:  The  analytic  signal  (AS)  is  a  positive  quantity  derived  from  magnetometer  data. 
It  can  easily  be  thresholded  for  target  detection  and  is  commonly  used  by  commercial 
practitioners.  The  analytic  signal  is  the  square  root  of  the  sum  of  the  gradients  in  three 
directions: 


AS  = 


where  the  z-gradient  is  formed  from  the  data  in  the  (x,y)  plane  by  upward  continuation  in  the 
Fourier  domain: 


=  kB(k ),  k  =  2nl  A  . 
dz 

Routines  to  calculate  the  analytic  signal  are  included  with  appropriately  licensed  versions  of 
Oasis  montaj™. 

Once  the  analytic  signal  is  calculated,  the  user  must  identify  individual  anomalies  by  picking 
peaks  above  the  background  noise.  The  spatial  extent  of  each  anomaly  can  be  estimated  by 
examining  the  nature  of  the  anomaly  in  terms  of  its  inflection  points  or  change  in  total  energy. 


Manual  Anomaly  Selection :  As  a  baseline  measure,  we  manually  identify  individual  anomalies 
and  estimated  the  spatial  footprint  of  each  anomaly  using  an  experienced  analyst  from  NAEVA 
Geophysics.  The  analyst  was  asked  to  select  all  anomalies  above  the  noise  level  of  the  given 
dataset.  The  analyst  decided  on  the  noise  threshold  and  properly  documented  the  rationale 
behind  the  threshold.  The  analyst  was  free  to  use  their  preferred  analysis  environment. 


2.2  Previous  Testing  of  the  Technology 

All  the  routines  that  we  evaluated  have  been  previously  developed  and  tested  commercially  as 
part  of  a  previous  research  program.  We  also  ran  shakedown  tests  on  the  four  algorithms  using  a 
synthetic  magnetic  dataset  with  60  well-characterized  targets  and  noise  characteristics  as  shown 
in  Figure  1.  The  synthetic  dataset  consisted  of  six  target  sizes  (20mm,  40mm,  60mm,  81mm, 
105mm  and  155mm)  randomly  placed  at  depths  up  to  the  1 1  times  the  target  diameter. 
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Figure  1.  Synthetic  magnetic  data  set  with  60  targets. 


2.2.1  Wavelet-based  Algorithm 


The  AWD  algorithm  has  been  successfully  applied  at  the  University  of  British  Columbia  as  part 
of  the  work  funded  by  Engineer  Research  and  Development  Center  (ERDC)  (DAAD 19-00-0- 
0120).  The  algorithm  was  demonstrated  on  three  datasets  with  different  characteristics  (Billings 
and  Herrmann,  2003).  In  each  case,  the  method  rapidly  located  the  majority  of  dipole  anomalies 
and  produced  accurate  estimates  of  the  dipole  parameters. 


There  were  two  main  enhancements  made  to  the  AWD  algorithm  under  this  project: 

1)  The  algorithm  was  wrapped  up  in  a  segmentation  routine  so  that  grids  of  any  size  can  be 
processed;  and 

2)  The  algorithm  was  implemented  within  the  SkyNET  software  environment  so  that  the 
original  Matlab  algorithm  could  be  seamlessly  accessed  from  with  Oasis  montaj™. 
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2.2. 1 . 1  Segmentation  Algorithm 

The  original  Matlab  implementation  of  the  AWD  algorithm  was  computationally  efficient  but 
required  a  lot  of  memory.  Once  the  grid  exceeded  a  size  of  about  1000  by  1000  points,  the 
algorithm  would  slow  down  as  Matlab  had  to  use  swap  space  to  manage  memory.  There  is  no 
need  to  use  the  entire  grid  to  detect  individual  targets  with  the  AWD.  Therefore,  we  developed  a 
segmentation  routine  so  that  grids  of  any  size  could  be  processed  with  the  algorithm. 

The  segmentation  algorithm  breaks  the  input  grid  up  into  sub-regions  comprising,  at  most,  500 
by  500  pixels.  The  region  is  then  buffered  by  50  pixels  to  avoid  edge-effects.  Targets  are  then 
detected  in  the  sub-region  and  its  buffer  zone  using  the  AWD  algorithm.  Any  detected  targets 
within  the  buffer  zone  are  rejected  as  these  will  also  be  detected  within  the  adjacent  sub-region. 
Figure  2  shows  an  example  of  the  sub-regions  and  the  associated  buffer-zones.  Using  this 
routine,  we  have  processed  grids  with  up  to  4000  by  4000  pixels. 
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Figure  2.  Segmentation  algorithm  applied  to  a  large  HeliMag  dataset.  The  solid  blue  lines 
delineate  the  boundaries  between  adjacent  sub-regions.  The  dashed  lines  delineate  the  buffer 
regions. 


2.2. 1.2  Integration  with  Oasis  Montaj™ 

The  AWD  algorithm  was  written  in  Matlab  code  and  is  accessible  from  the  UXOLab  software 
environment  (UBC  and  Sky’s  UXO  analysis  software).  Rather  than  porting  the  algorithm  to  C++ 
and  wrapping  it  within  a  GX  (a  laborious  and  time-consuming  process)  we  decided  to  develop  a 
mechanism  to  call  the  Matlab  algorithm  directly  from  Oasis  montaj™.  This  was  achieved  by 
implementing  the  algorithm  within  SkyNet,  Sky  Research’s  custom  designed  .NET  based 
analysis  software. 

SkyNet  consists  of  several  .NET  dlls  that  can  be  linked  to  other  .NET  compatible  applications, 
such  as  Oasis  montaj™.  SkyNet  also  uses  the  COM  interop  mechanism  of  .NET  to  expose  its 
functionality  to  other  applications,  such  as  MatLab,  that  do  not  inherently  support  .NET.  As  a 
result,  there  are  three  different  ways  to  run  SkyNet: 

1)  Running  SkyNet  from  within  the  Oasis  montaj™  application  process 

2)  Running  SkyNet  from  within  the  MatLab  application  process 

3)  Running  SkyNet  by  itself  (from  within  the  SkyNet  application  process) 

For  the  Wavelet  algorithm  we  used  option  1  and  ran  everything  from  within  the  Oasis  montaj™ 
process.  Figure  3  below  provides  an  overview  of  this  configuration: 


Legend 

(  )3rd  party  software 
□□  Sky  code 
C#,  VB.NET 
MatLab  code 
C++  code 


Figure  3.  Schematic  of  the  SkyNet  architecture. 
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As  shown  in  Figure  3,  all  SkyNet  DLLs  run  within  the  Oasis  montaj™  executable  process.  At 
startup,  an  Oasis  montaj™  menu  item  calls  the  SkyNet  COM  and  GUI  DLL  that  will  in  turn 
initialize  the  compiled  MatLab  code  DLL.  During  the  initialization,  a  COM  event  callback  is 
registered  in  the  compiled  MatLab  code  DLL  that  allows  SkyNet  to  call  MatLab  functions.  This 
also  allows  the  MatLab  function  to  ‘Set’  and  ‘Get’  data  to  and  from  SkyNet.  The  Geosoft  API  is 
linked  to  the  Base  Classes  DLL,  allowing  SkyNet  to  interact  with  the  Oasis  montaj™  application 
GUI.  Once  all  initialization  is  complete,  the  SkyNet  menu  item  in  Oasis  montaj™  can  utilize 
any  SkyNet  DLL  functionality  (i.e.  all  functions  found  in  C#,  VB.NET,  and  MatLab  compiled 
code).  Figure  4  below  shows  the  AWD  algorithm  being  called  directly  from  Oasis  montaj™.  The 
user  experience  is  almost  identical  to  that  for  any  other  Geosoft  algorithm:  the  only  noticeable 
variation  is  that  the  parameter  input  GUI  has  a  slightly  different  appearance. 

To  access  the  algorithm,  the  user  needs  to  install  the  Matlab  run-time  libraries  and  the  SkyNET 
application  on  his/her  PC.  Both  of  these  applications  have  been  bundled  into  a  standard 
installation  package  so  that  no  special  knowledge  is  required  for  the  installation.  The  user  then 
installs  the  SkyNet  menu  through  a  standard  OMN  file  that  is  a  standard  menu-configuration 
process  for  any  new  Geosoft  application.  Note  that  the  Matlab  run-time  library  can  be  installed 
free  of  charge. 

At  present  the  main  limitation  is  that  SkyNet  is  only  compatible  with  Geosoft  Version  6.2  (the 
current  version  is  6.4).  This  is  because  SkyNet  currently  uses  the  .NET  Version  1.1  whereas 
Geosoft  6.4  uses  .NET  Version  2.0.  There  are  a  number  of  incompatibilities  between  these  two 
versions  of  .NET.  Later  this  year,  SkyNet  will  be  upgraded  to  .NET  version  2.0  and  the  AWD 
algorithm  will  then  be  accessible  from  Geosoft  6.4  and  higher.  If  Geosoft  then  moves  to  .NET 
Version  3.0,  the  algorithm  should  still  be  accessible  because  .NET  is  now  configured  to  be 
backwards  compatible  (except  with  version  1.1). 
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Figure  4.  Screen-capture  of  the  AWD  algorithm  being  used  directly  from  within  Oasis  montaj™. 


2.2. 1.3  Synthetic  Magnetic  Data  Test 

The  AWD  algorithm  was  applied  to  the  synthetic  magnetic  data  and  found  43  of  the  60  targets 
with  25  false  positives  as  shown  in  Figure  5.  By  changing  one  of  the  key  parameters  of  the 
wavelet,  3  additional  targets  were  found  but  at  the  expense  of  55  additional  false  positives. 
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Figure  5.  Results  of  AWD  algorithm  on  the  synthetic  magnetic  data.  The  60  true  locations  are 
marked  as  white  circles  and  the  wavelet  picks  as  black  crosses. 


2.2.2  Rule-based  Clusters 

The  two  rule-based  clustering  algorithms  that  were  used  as  a  basis  for  the  clustering  algorithm 
developed  for  this  project  were  previously  tested  during  their  development  as  part  of  the  work 
funded  by  SERDP  (CU-1092)  and  NAVEODTECHDIV.  The  AETC  Incorporated  clustering 
algorithm  was  also  tested  on  data  collected  at  the  Badlands  Bombing  Range  (BBR)  on  the  Pine 
Ridge  Reservation  in  South  Dakota  and  at  29  Palms  in  California  (Barrow,  1998).  The  results  of 
this  test  showed  the  automatic  processor  detecting  about  90-100%  of  the  objects  present  with  the 
most  significant  detection  factor  being  target  density.  In  low  density  regions  closer  to  100%  of 
the  objects  were  found  whereas  in  high  density  areas  the  detection  rate  fell  to  90%.  It  was  also 
estimated  the  automatic  processor  could  analyze  about  4  times  more  data  than  the  manual 
processor  in  the  same  amount  of  time. 

We  reviewed  both  codes  in  detail  and  incorporated  unique  and  attractive  capabilities  from  each. 
We  essentially  used  the  clustering  algorithm  developed  by  AETC  Incorporated  and  the 
association  algorithm  developed  by  Blackhawk.  After  testing  the  clustering  method  picker  on  a 
suite  of  synthetic  test  dipoles  and  comparing  the  inverted  parameters  to  those  obtained  by  manual 
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target  selection,  we  determined  the  circumstances  in  which  the  automatic  picker  performed 
poorly  and  made  the  modification  to  the  code  as  outlined  below. 

One  recurring  situation  was  the  loss  of  a  lobe  due  to  the  initial  signal  thresholding.  Since 
lowering  this  threshold  too  much  would  result  in  noise  being  flagged  as  targets,  an  alternate 
method  was  developed.  After  all  positive  and  negative  dipoles  have  been  combined,  the  code 
examines  single  dipoles,  and  searches  for  points  of  the  opposite  sign  above  a  lower  extraction 
threshold,  within  a  certain  distance  from  the  peak,  and  such  that  the  putative  dipole  has  a 
declination  within  a  certain  offset  of  the  Earth's  field  declination.  This  method  successfully 
recovered  the  lost  lobe  in  most  cases,  and  improved  the  inverted  target  parameters.  The  code  also 
tries  to  flag  potentially  overlapping  targets  by  measuring  the  gradient  change  as  one  moves  out 
from  a  dipole  peak.  These  cases  can  be  examined  manually  by  the  user.  The  original  version  was 
far  too  sensitive,  however,  with  noise  often  resulting  in  flags  where  no  target  overlap  existed. 

The  code  was  modified  to  add  an  additional  requirement  that  an  already  determined  dipole  must 
exist  within  a  certain  distance  and  angle  of  the  gradient  change  location.  This  resulted  in  a 
significant  decrease  of  the  spurious  overlap  flags. 

The  development  and  testing  of  these  additions  were  completed  using  IDL.  The  final  version 
was  recoded  into  C++  and  seamlessly  integrated  into  Oasis  montaj™  through  a  custom  GX.  The 
GX  wrapper  was  written  in  the  Microsoft  .NET  developer  environment  which  allows  the 
developer  to  build  single  functions  or  suites  of  functions  into  a  .NET  DLL  that  is  run  directly 
from  Oasis  montaj™.  During  the  integration  to  Oasis  montaj™  an  option  to  output  an  ascii  file 
that  is  compatible  with  UX-Analyze  (ESTCP  funded  modeling  and  visualization  software  that 
runs  under  the  Oasis  montaj™)  was  added.  The  ascii  file  contains  the  geographic  boundary 
coordinates  that  defines  the  spatial  footprint  of  the  picked  anomaly.  The  ‘polygon’  is  a  circle 
that  encompasses  all  the  points  in  the  positive  and  negative  clusters  that  are  associated  with  the 
anomaly.  The  radius  of  this  circle  is  increased  by  20%  to  include  background  values  around  the 
anomaly  which  are  a  valuable  input  to  UX-Analyze. 

As  part  of  this  demonstration  the  clustering  algorithm  was  applied  to  the  synthetic  magnetic  data 
set  and  found  46  of  the  60  targets  with  1 1  false  positives  as  shown  in  Figure  6. 
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Figure  6.  Results  of  clustering  algorithm  on  the  synthetic  magnetic  data.  The  true  locations  are 
plotted  as  circles  and  the  automatic  picker  detections  are  plotted  as 


2.2.3  Dipole  based  Matched  Filter 


Previous  testing  of  the  MF  algorithm  included  MTADS  magnetometer  survey  data  over  several 
sites,  which  include  29  Palms,  Blossom  Point  and  Buckley  Field.  The  algorithm  was  also  tested 
on  the  three  1-hectacre  areas  of  the  2000  Advanced  UXO  Detection/Discrimination  Technology 
Demonstration  at  JPG.  The  MF  did  a  good  job  at  isolating  potential  targets  at  JPG  (Bell,  2001). 
The  fdter  output  peaked  over  the  targets,  which  helped  reduce  the  ambiguity  of  the  exact  target 
location.  The  low  probability  of  detection  (80%)  was  caused  by  data  gaps  and  the  difficulty  in 
finding  20mm  using  magnetometer  data  with  a  sensor  spacing  of  25cm. 


As  part  of  ESTCP  project  199918,  the  MFAP  was  integrated  into  Oasis  montaj™  version  5.13  in 
the  year  2002.  Since  then  Oasis  montaj™  has  undergone  a  major  change  as  is  currently 
releasing  version  6.4.  The  upgrade  to  version  6  included  significant  enhancements  to  software 
usability,  licensing,  interface  and  map  editing  functions.  These  upgrades  created  an 
incompatibility  with  the  original  MFAP  GX.  The  main  issues  were  changes  to  the  structure  of 
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the  GX  wrapper  and  changes  to  the  syntax  and  parameters  of  some  functions  called  within  the 
GX.  These  incompatibilities  were  fixed  and  the  GX  was  recompiled  and  tested  in  versions  6.3 
and  6.4. 

As  part  of  this  demonstration  the  MF  algorithm  was  applied  to  the  synthetic  magnetic  data  set 
and  found  43  of  the  60  targets  with  46  false  positives  as  shown  in  Figure  7.  Many  of  the  false 
positives  were  the  result  of  multiple  declarations  for  a  single  target  because  of  processing 
artifacts  in  the  match  filter  output  that  are  caused  when  the  filter  window  contains  only  a  portion 
of  the  dipolar  signal. 
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Figure  7.  Color  contour  map  of  the  match  filter  output  calculated  from  the  synthetic  magnetic 
data.  The  true  locations  are  plotted  as  circles  and  the  automatic  picker  detections  are  plotted  as 
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2.2.4  Analytic  Signal 


The  analytic  signal  is  included  with  appropriately  licensed  versions  of  Oasis  montaj™  and  is 
commonly  used  by  commercial  analysts.  As  such,  it  has  undergone  internal  testing  of  the 
algorithm  at  Geosoft  and  extensive  practical  testing  by  the  user  community. 

The  analytic  signal  algorithm  produces  an  anomalous  surface  with  peaks  over  potential  targets 
but  does  not  output  a  ‘polygon’  file  containing  the  spatial  extents  of  the  targets.  As  mentioned 
earlier  this  polygon  file  is  an  essential  input  to  UX -Analyze.  For  this  reason,  a  GX  was 
developed  to  estimate  the  spatial  extents  and  output  the  boundary  coordinates  to  a  UX- Analyze 
compatible  file.  For  each  anomaly  location,  the  ‘footprint’  algorithm  uses  the  analytic  signal 
grid  and  starts  at  the  nearest  grid  cell  to  a  given  target  location.  The  algorithm  searches  along 
each  of  the  eight  directions  and  stops  if  one  of  the  following  three  conditions  is  met:  1)  one  of 
the  surrounding  grid  cells  away  from  the  peak  is  larger  in  amplitude,  2)  the  value  is  less  than  the 
user  defined  background  value,  and  3)  the  distance  from  the  initial  location  exceeds  the  user 
defined  limit.  Upon  completion  of  the  search,  there  are  eight  stopping  locations  surrounding  the 
anomaly.  At  this  point  the  user  has  the  option  to  output  these  points  to  a  file  or  to  output  a  circle 
to  the  file.  The  radius  of  the  circle  is  the  median  of  the  distances  from  the  anomaly  location  to 
each  of  the  eight  stopping  locations.  Areas  with  many  close  or  overlapping  anomalies  favor  the 
former  option  while  areas  with  isolated  anomalies  will  the  benefit  from  the  calculated  circle  as 
shown  in  Figure  8. 
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Figure  8.  Color  contour  map  of  the  analytic  signal  grid  calculated  from  the  synthetic  magnetic 
data.  The  calculated  footprint  of  selected  anomalies  using  the  polygon  (red)  and  circle  (black) 
options  are  shown. 

The  analytic  signal  algorithm  was  applied  to  the  synthetic  magnetic  data  to  create  an  analytic 
signal  grid.  The  grid  was  filtered  with  two  passes  of  a  9  point  hanning  filter  and  anomalies  were 
picked  using  the  Blakely  method  (Blakely,  1986).  The  Blakely  method  compares  the  value  of 
each  grid  cell  with  values  of  the  eight  (8)  nearest  grid  cells  in  four  directions  (along  the  row, 
column,  and  both  diagonals).  If  the  grid  value  is  greater  than  the  input  threshold  and  all  the 
nearest  grid  cells  are  lower,  it  is  selected  as  a  peak.  Using  a  threshold  of  45nT/m,  49  of  the  60 
targets  were  found  with  no  false  positives  as  shown  in  Figure  9. 
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Figure  9.  Color  contour  map  of  the  analytic  signal  grid  calculated  from  the  synthetic  magnetic 
data.  The  true  locations  are  plotted  as  circles  and  the  automatic  picker  detections  are  plotted  as 


2.3  Factors  Affecting  Cost  and  Performance 

Implementation  of  the  automatic  picking  methods  used  in  this  Demonstration  should 
considerably  reduce  the  time  and  thus  cost  required  to  pick  anomalies  when  compared  to  the 
manual  method.  It  is  anticipated  this  will  be  the  case  in  areas  with  low  anomaly  density  but  as 
the  anomaly  density  increases  to  the  point  that  overlapping  anomalies  are  prevalent,  the 
performance  of  the  automatic  pickers  may  be  degraded.  Also,  the  quality  of  the  data  would  have 
an  impact  on  the  performance  of  the  automatic  methods.  If  the  data  is  sufficiently  noisy,  the 
automatic  methods  will  likely  either  pick  too  many  targets  if  the  threshold  is  set  to  capture 
smaller  anomalies  or  miss  the  small  anomalies  if  the  threshold  is  set  above  the  background  noise 
level.  Whereas  the  analyst  using  the  manual  method  may  be  able  to  screen  out  the  anomalies 
caused  by  noisy  data  by  using  his  experience  and  the  “look”  of  the  anomaly. 
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2.4  Advantages  and  Limitations  of  the  Technology 


This  demonstration  used  already  collected  survey  data  to  test  four  automatic  target  pickers  and 
transitioned  the  algorithms  to  the  user  community  through  Oasis  montaj™.  The  main  advantage 
of  using  automatic  methods  to  pick  anomalies  is  the  time  it  can  save.  On  large  remediation 
projects  with  thousands  of  anomalies,  the  manual  method  is  very  slow.  Another  aspect  of  the 
manual  method  is  its  subjective  nature.  Different  data  analysts  may  have  different  criteria  on 
what  is  and  is  not  an  anomaly.  It  may  be  based  on  signal  level  and  signal  shape  which  is  directly 
affected  by  the  way  the  data  is  displayed.  The  target  density  of  the  data  may  also  affect  the 
analyst’s  decision.  In  areas  of  low  density,  the  analyst  may  be  prone  to  pick  smaller  and  weaker 
anomalies  for  the  sake  of  picking  something.  Conversely,  in  high-density  areas  the  analyst  may 
set  his  threshold  higher.  Based  on  these  two  difficulties,  automatic  target  pickers  are  desirable 
because  they  would  only  be  limited  by  the  speed  of  the  computer  and  can  be  set  to  select  targets 
using  fixed  criteria. 

Previous  testing  of  the  automatic  pickers  has  shown  limitations  to  the  technology.  The  clustering 
and  analytic  signal  algorithms  may  have  trouble  with  low  signal-to-noise  signatures  and  high 
background  noise.  In  these  areas,  a  low  threshold  must  be  chosen  to  pick  the  weaker  anomalies 
but  by  lowering  the  threshold,  the  automatic  pickers  will  pick  many  anomalies  that  are  caused  by 
noise  in  the  data. 

The  matched  filter  algorithm  cannot  filter  a  point  if  there  are  any  missing  data  in  the  filter  box. 
Thus,  the  edges  of  a  survey  site  cannot  be  filtered  as  well  as  any  interior  regions  that  are  missing 
data.  Therefore,  a  small  data  gap  of  0.5x5m  becomes  an  unfilterable  region  of  3.5x8m  if  a  3m 
filter  box  is  used.  For  this  reason  the  match  filter  algorithm  cannot  be  used  on  the  WAA 
vehicular  transect  data  because  the  width  of  the  transect  is  only  1.75m. 

The  analytic  signal  and  AWD  algorithms  map  and  interpolate  the  magnetic  data  to  a  regular  x-y 
grid  and  apply  their  respective  filters  to  the  grid.  For  normal  surveys,  this  would  not  be  an  issue 
but  applying  them  to  transect  data  may  require  some  optimization.  Because  transects  can  cover 
large  areas  in  a  single  sortie  the  resultant  grid  can  get  very  large  given  that  the  grid  produced  is 
based  on  the  minimum  and  maximum  XY  values.  If  the  transect  lines  are  relatively  straight 
along  one  geographic  heading,  the  data  can  be  rotated  prior  to  creating  the  grid  such  that  the  lines 
are  oriented  along  either  the  grid  rows  or  columns. 

The  analytic  signal  and  matched  filter  algorithms  output  an  anomalous  surface  and  not  the 
locations  of  the  anomalies.  Therefore,  a  second  program  must  be  run  to  pick  the  peaks  in  the 
surfaces.  Thus,  the  success  of  these  methods  is  partially  dependent  on  the  peak  picking  program. 

The  clustering  and  AWD  algorithms  have  several  parameters  that  need  to  be  set.  The  option  to 
change  many  of  the  parameters  allows  the  user  the  flexibility  to  tune  the  algorithms  to  the 
individual  data  characteristics.  However,  this  flexibility  adds  to  the  complexity  and  setup  time  to 
ensure  the  proper  parameters  are  picked  to  achieve  optimum  perfonnance  of  the  algorithms. 
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3.  Demonstration  Design 


3.1  Performance  Objectives 

Table  1  and  Table  2  outline  the  perfonnance  objectives  for  the  production  and  WAA  surveys, 
respectively.  The  values  of  the  quantitative  performance  objectives  for  the  production  surveys 
were  chosen  to  measure  how  practical  the  methods  are  for  use  in  the  “real”  world.  The  picks  by 
the  different  methods  were  compared  to  the  ground  truth  for  the  production  surveys  by 
calculating  the  Probability  of  detection  (Pd)  defined  as  (#  of  detections  picked  by  an  automatic 
picker  that  are  coincident  to  a  ground  truth  object  /  #  of  ground  truthed  objects).  The 
Background  alarm  rate  (rBAR)  defined  as  (#  of  detections  picked  by  automatic  picker  not  caused 
by  ground  truthed  object  /  arbitrary  number  (in  order  to  conceal  the  absolute  number  of  truth 
items))  was  calculated  only  for  the  APG  data  set.  This  was  because  none  of  the  other  data  sets 
contained  complete  ground  truth  so  the  additional  picks  may  or  may  not  have  been  caused  by  a 
valid  object.  Because  the  calculation  used  an  arbitrary  number  the  actual  performance  metric  is 
meaningless  but  useful  information  is  gained  by  comparing  the  number  among  the  different 
target  picking  methods.  The  manual  method  was  used  as  the  baseline  to  judge  the  efficiency  of 
the  different  methods.  Our  goal  is  for  the  processing  time  (time  required  to  run  automatic  picker 
/  time  required  to  manually  pick  the  anomalies)  to  be  less  than  0.25.  This  would  allow  plenty  of 
time  to  manually  alter  the  automatic  picks  to  achieve  the  best  possible  performance  and  still 
complete  the  analysis  in  less  time  than  the  using  only  the  manual  method. 

We  compared  the  fitted  parameters  using  results  from  the  automatic  methods  and  the  manual 
method  to  ground  truth  information  for  80  targets  at  APG.  The  XY  location  and  a  boundary  file 
that  estimates  the  anomaly’s  spatial  extent  were  used  as  inputs  to  advanced  physics  based 
inversion  routines  that  output  target  features  such  as  XY  location,  depth  and  size.  The  detection 
location  accuracy  is  defined  as  the  average  horizontal  distance  between  the  picked  location  that 
is  output  directly  from  the  target  picking  algorithms  and  the  ground  truth  location.  The 
characterization  location  accuracy  is  the  average  horizontal  distance  between  the  XY  position 
output  from  the  inversion  routines  and  the  ground  truth  location.  Therefore,  assuming  good  data 
quality  the  characterization  location  should  be  more  accurate  than  the  detection  location.  The 
characterization  size  accuracy  is  defined  as  (the  absolute  difference  between  the  fitted  target 
diameter  and  the  ground  truth  diameter)/(ground  truth  diameter).  The  location,  depth  and  size 
metrics  were  set  to  values  that  would  enable  the  dig  teams  to  easily  find  the  target. 
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Table  1.  Performance  Objectives  for  production  surveys 


Type  of 
Performance 
Objective 

Primary 

Performance 

Criteria 

Expected 

Performance 

(metric) 

Actual  Performance 

Qualitative 

Ease  of  use 

General  Observations 

Some  training  required  to  set 
proper  parameters  for  data 
processor  experienced  in  Oasis 
montaj™ 

Robustness 

General  Observations 

Analysis  flow  not  seriously 
interrupted  by  bugs 

Quantitative 

Probability  of 
Detection  (ground 
truth) 

>0.9 

0.15-0.99 

Background  Alarm 
Rate  (APG  only) 

NA 

0.105-0.615 

Setup  Time 

<4hours 

0.1  -  5.45  hours 

Processing  Time 

<0.25 

.02-.25 

Detection  Location 
Accuracy 

90%  <0.5m 

63-95% 

Characterization  Size 
Accuracy  (APG  only) 

90%  <0.3 

39-44% 

Characterization 
Location  Accuracy 
(APG  only) 

90%  <0.3m 

90-93% 

Characterization 

Depth  Accuracy 
(APG  only) 

90  %  <0.3m 

94-97% 

The  WAA  surveys  have  the  similar  performance  criteria  as  the  production  surveys  but  because  of 
differing  objectives  the  expected  performance  will  be  different.  The  general  objective  of  the 
WAA  surveys  is  to  delineate  munitions  response  sites,  support  regulatory  disposition  of  non 
munitions  response  sites  and  provide  reliable  data  to  support  risk  analysis  and  remedial  cost 
estimation.  The  probability  of  detection  metric  is  lower  because  of  the  different  objectives.  To 
achieve  these  goals  airborne  magnetometer  data  and  vehicle  based  magnetic  transect  data  were 
collected.  These  data  sets  have  much  lower  data  densities  which  results  in  the  location  metrics 
being  higher  than  for  production  surveys.  Also,  ground  truth  data  was  not  available  for  the  WAA 
data  sets  so  the  Pd  and  Pfa  were  calculated  using  the  manual  picks  as  the  baseline. 
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Table  2.  Performance  Objectives  for  WAA  surveys 


Type  of 
Performance 
Objective 

Primary 

Performance 

Criteria 

Expected 

Performance 

(metric) 

Actual  Performance 

Qualitative 

Ease  of  use 

General  Observations 

Some  training  required  to  set 
proper  parameters  for  data 
processor  experienced  in  Oasis 
montaj™ 

Robustness 

General  Observations 

Analysis  flow  not  seriously 
interrupted  by  bugs 

Quantitative 

Probability  of 
Detection  (manual 
declarations) 

>0.8 

0.49-0.76 

Probability  of  False 
Alarm  (manual 
declarations) 

<0.25 

0.02-0.90 

Setup  Time 

<4hours 

0.15-5.08  hours 

Processing  Time 

<0.25 

0.04-0.26 

Detection  Location 
Accuracy 

90%  <  1.5m 

92-100% 

3.2  Selecting  Test  Sites 

This  demonstration  was  performed  in  the  Cary,  NC  office  of  SAIC  (formerly  AETC)  on  data  that 
was  previously  collected  on  government  ranges  for  other  purposes.  There  are  thus  no  regulatory, 
health  or  safety  issues  affecting  this  demonstration. 

The  COE  was  asked  to  provide  examples  of  data  sets  that  are  representative  of  that  acquired  by 
their  contractors  and  that  support  our  requirements.  Our  criteria  for  the  demonstration  data  sets 
were  the  following: 

target  type,  size  and  density.  A  range  of  data  density  and  target  sizes,  and  a  variety  of  target 
types  were  needed  to  explore  the  capabilities  and  perfonnance  of  the  four  automatic  pickers. 

signal  and  noise  levels  and  signal  to  noise  ratio  (SNR).  Picking  anomalies  can  be  relatively  easy 
if  the  SNR  is  extremely  high,  but  can  be  very  difficult  if  the  SNR  is  extremely  low  (since 
identifying  the  positive  and  negative  lobe  of  individual  anomalies  can  be  impossible).  Thus,  we 
desired  a  range  of  SNR  from  marginal  to  excellent  in  order  to  derive  conclusions  that  can  be 
extrapolated  for  future  use.  This  same  reasoning  applied  to  the  requirements  for  noise 
characteristics  and  data  density  as  well. 
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geologic  noise  levels.  Data  sets  should  represent  a  range  of  geologic  noise  from  low  to  high. 

This  is  critical  to  being  able  to  say  that  the  results  of  this  demonstration  are  applicable  for  COE 
contractors  in  general. 

platform.  The  target  pickers  were  demonstrated  on  data  collected  from  different  platforms, 
including  vehicular,  man  portable  and  airborne  (e.g.,  helicopter-towed  data  from  the  NRL 
system) 

data  quality.  Data  sets  were  obtained  that  are  of  both  Standard  Contractor  and  Research  quality 
in  terms  of  sensor  noise,  lag,  etc. 

site  location.  The  data  sets  should  not  all  be  from  the  same  site. 

number  of  targets.  Each  data  set  should  contain  at  least  100,  but  ideally  no  more  than  500, 
anomalies. 

available  ground  truth.  Because  this  proposed  work  was  primarily  concerned  with  the  detection 
of  all  anomalies  above  the  sensor  noise  floor  whether  they  are  UXO  or  not  and  not  the 
discrimination  of  the  anomalies,  ground  truth  information  was  desirable  for  all  data  sets  but  not 
absolutely  necessary.  When  available,  we  focused  our  analysis  on  comparing  the  anomaly  picks 
to  the  ground  truth  information.  If  this  information  was  not  available,  we  used  the  manual  picks 
as  the  baseline  for  comparison  to  the  automatic  picks.  Accurate  ground  truth  infonnation 
(location,  burial  depth,  size,  and  description)  was  needed  for  50-100  targets  in  order  to  compare 
the  location  accuracy  and  the  extracted  spatial  footprint  of  the  anomaly  for  the  different  picking 
algorithms. 

In  addition  to  the  data  sets  provided  by  the  COE,  the  program  office  requested  that  we  also 
include  helicopter  data  and  ground  based  transect  data  from  the  ESTCP  WAA  Pilot  Program. 

We  also  ran  the  target  pickers  on  a  few  research  grade  data  sets  that  met  our  requirements.  The 
following  briefly  describes  each  of  the  data  sets. 

Three  of  the  data  sets  were  acquired  using  the  vehicle  towed  MTADS  by  NRL  and  AETC 
Incorporated.  The  MTADS  hardware  consists  of  a  low  magnetic  signature  vehicle  that  is  used  to 
tow  an  array  of  eight  magnetic  sensors  that  are  spaced  .25  meters  apart.  The  sensor  positions 
were  determined  using  real-time  kinematic  (RTK)  GPS  receivers.  The  three  data  sets  were 
collected  at;  1)  Open  field  grid  of  the  Aberdeen  Proving  Ground  Standardized  test  site  in  June 

2004,  2)  Target  SI  on  the  Pueblo  of  Isleta  near  Albuquerque,  NM  in  February  2003,  and  3)  Area 
near  Bombing  Target  4  (BT-4)  at  Pueblo  Precision  Bombing  Range  in  Colorado  in  September 

2005. 

Another  data  set  was  collected  by  Sky  Research  and  AETC  Incorporated  using  the  airborne 
MTADS  system.  The  system  hardware  includes  an  array  of  seven  magnetometers  spaced  1.5 
meters  apart  in  a  9  meter  boom  mounted  on  a  Bell  206L  Helicopter.  The  sensor  positions  were 
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determined  using  RTK  GPS  technology.  The  data  collected  was  in  the  same  vicinity  as  the 
above-mentioned  vehicle  towed  MTADS  data  near  BT-4  at  Pueblo  Precision  Bombing  Range. 

USAGE  contractors  acquired  the  remaining  three  data  sets.  American  Technologies 
Incorporated  acquired  data  on  a  removal  action  over  a  300-acre  area  at  JPG.  The  data  were 
collected  using  the  handheld  Geometric  G-858G  vertical  gradiometer  system  in  fiducial 
positioning  mode.  Two  magnetometers  were  vertically  separated  by  2  feet  and  data  were 
collected  using  a  lane  spacing  of  2.5  feet.  NAEVA  Geophysics  collected  magnetic  data  over  a 
Geophysical  Proveout  (GPO).  (The  USAGE  has  requested  that  the  site  specifics  be 
confidential.)  The  data  were  collected  using  a  single  handheld  magnetometer  with  a  line  spacing 
of  2  feet  and  positioned  using  RTK  GPS.  The  last  data  set  was  collected  by  Parsons  over  the 
Seaside  area  at  the  Former  Fort  Ord.  The  data  were  collected  using  a  handheld  array  of  four 
magnetometers  spread  cross  line  at  2-foot  intervals  and  positioned  using  RTK  GPS. 

Six  of  these  seven  data  sets  had  at  least  100  anomalies  and  a  few  exceeded  the  500  anomaly  limit 
which  increased  our  analysis  time  but  were  beneficial  to  the  demonstration  (in  the  case  of  very 
large  data  sets,  we  selected  only  a  portion  to  analyze).  The  one  data  set  (US ACE  GPO  data)  with 
fewer  than  100  anomalies  was  chosen  because  we  had  a  limited  number  of  datasets  (three)  that 
were  representative  of  data  acquired  by  USAGE  contractors. 

An  action  item  for  Project  MM- 1455  following  the  6  February  IPR  made  the  following  request: 

The  Program  Office  is  interested  in  a  comparison  between  this  method  and  the 
auto  pickers  that  are  being  evaluated  in  the  AETC  ESTCP  project  “ Target 
Picking  Methods  for  Magnetic  Data,  MM-0502.  ”  ....select  one  or  more  common 
datasets. 

After  discussions  with  Jim  McDonald,  it  was  decided  that  the  vehicular  data  set  from 
Isleta  and  the  vehicular  data  set  from  APG  were  the  two  best  candidates  for  the 
comparison.  Additional  information  can  be  found  in  a  White  Paper  submitted  by  Jim 
McDonald  to  the  Program  Office  addressing  the  action  item  above  for  project  MM- 145 5. 
Following  discussions  with  Anne  Andrews,  it  was  decided  that  the  Isleta  data  set  would 
be  used  for  the  comparison  conducted  under  project  MM-1455. 

Figure  10  to  Figure  16  shows  the  data  as  color  contour  maps  with  ground  truth  locations  overlaid 
with  circles  or  crosses  for  each  of  the  data  sets.  Table  3  below  summarizes  how  well  the  data 
sets  fit  our  demonstration  criteria. 
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Table  3.  Criteria  and  Data  sets 


data  set/ 
criteria 

NRL 
MTADS 
APG  June 
2004 

USAGE 

JPG 

USAGE 

GPO  data 

WAA 

Pueblo 

BT4 

ground 

transects 

WAA 

Pueblo 

BT4 

airborne 

Isleta 

vehicle 

Seaside 

target- 

sizes 

20mm 

40mm 

57mm 

60mm 

81mm 

2.75in 

105mm 

155mm 

bdu28 

blu26 

M42 

MK118 

OE  scrap 
4.5in  head, 
mortar 

burster 

MKIIgrenade 
MKII17#bomb 
MKI  25#bomb 
range  clutter 

M3  8 

MKI  5 

GP  Bomb 

M3  8 

MKI  5 

GP  Bomb 

60mm 

81mm 

2.75in 

105mm 

BDU33 

MK76 

M3  8 

GP  Bomb 

37  mm, 
60mm, 

3.5in  M29, 

Grenade 

fuze, 

Non  OE 
scrap 

target 

density 

low-high 

low-high 

low-med 

low-med 

low-med 

low-med 

low-high 

platform 

vehicle 

handheld 

handheld 

vehicle 

airborne 

vehicle 

handheld 

data 

quality 

high 

[research] 

low 

[standard] 

med 

[standard] 

high 

[research] 

high 

[research] 

high 

[research] 

med 

[standard] 

SNR 

low-high 

low-high 

low-high 

low-high 

low-high 

low-high 

low-high 

geologic 

noise 

low-med 

low 

low-high 

low-med 

med-high 

low-med 

low-med 

number  of 
ground 
truthed 
anomalies 

All  but  not 
released 
to  public 

253 

33 

2 

0 

150 

481 

Survey 

objective 

production 

production 

production 

WAA 

WAA 

production 

production 

Picking 

methods 

AWD,  AS, 

Matched 

filter, 

Clustering, 

Manual 

AWD,  AS, 

Matched 

filter, 

Clustering, 

Manual 

AWD,  AS, 

Matched  filter, 
Clustering, 
Manual 

AWD,  AS, 
Clustering, 
Manual 

AWD,  AS, 

Matched 

filter, 

Clustering, 

Manual 

AWD,  AS, 

Matched 

filter, 

Clustering, 

Manual 

AWD,  AS, 

Matched 

filter, 

Clustering, 

Manual 
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Figure  10.  NRL  MTADS  magnetic  data  collected  at  APG  Open  field.  The  red  box  outlines  the 
area  evaluated  by  the  target  pickers.  The  crosses  show  publicly  released  ground  truth. 
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Figure  1 1.  USACE  magnetic  data  collected  at  Jefferson  Proving  Ground.  The  crosses  show 
known  ground  truth. 
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Figure  12.  US  ACE-supplied  magnetic  data  over  GPO.  The  crosses  show  known  ground  truth. 
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Figure  13.  Airborne  magnetic  data  from  near  BT4  at  the  Pueblo  Precision  Bombing  Range  in 
Colorado. 
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Figure  14.  Vehicular  magnetic  data  from  Isleta  South.  The  crosses  show  known  ground  truth. 
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Figure  15.  Magnetic  data  from  Seaside.  The  crosses  show  known  ground  truth. 
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Figure  16.  This  map  shows  the  location  of  the  vehicle  based  magnetic  transect  data  near  BT4  at 
the  Pueblo  Precision  Bombing  Range  in  Colorado.  The  red  lines  show  the  transects  included  in 
this  demonstration. 
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3.3  Test  Site  History/Characteristics 


The  area  of  the  Aberdeen  Test  Site  is  adjacent  to  the  Trench  Warfare  facility  at  the  APG.  The 
APG  Standardized  Test  Site  is  located  within  a  secured  range  area  of  the  Aberdeen  Proving 
Grounds.  The  Aberdeen  Area  of  APG  is  located  approximately  30  miles  northeast  of  Baltimore 
at  the  northern  end  of  the  Chesapeake  Bay.  The  Standardized  Test  Site  encompasses  17  acres  of 
upland  and  lowland  flats,  woods  and  wetlands.  The  Test  Site  is  divided  into  areas  including 
calibration  lanes,  blind  grid  test  grid,  open  field,  mogul,  and  wooded  areas.  Additional  details 
regarding  the  layout  of  the  APG  Standardized  Test  Site  can  be  found  at 
http://aec.anny.mil/usaec/technology/uxo03c0 1  .html. 

The  former  Pueblo  Precision  Bombing  and  Pattern  Gunnery  Range  #2  consists  of  a  total  of 
67,769  acres  and  is  located  approximately  20  miles  south  of  La  Junta,  Colorado,  in  Otero 
County.  The  closest  community  is  La  Junta,  a  rural  town  with  a  population  of  about  7,637.  The 
MRA  was  used  by  local  populations  for  cattle  grazing  until  the  War  Department  assumed  control 
of  the  lands  to  construct  the  Pueblo  Precision  Bombing  and  Pattern  Gunnery  Range  #2  (1942  to 
1946). 

The  Pueblo  of  Isleta  is  located  in  north-central  New  Mexico,  approximately  10  miles  south  of 
Albuquerque.  The  reservation  is  bordered  on  the  north  by  the  Sandia  Military  Reservation, 
which  includes  Kirtland  Air  Force  Base,  the  Manzano  Mountains  on  the  east,  and  the  Rio  Puerco 
and  Laguna  Pueblo  Reservation  on  the  west.  The  area  that  contains  target  S 1  comprises  an  area 
of  approximately  7000  acres  that  were  leased  from  the  Tribe  in  the  1950’s  for  use  as  a  target 
bombing  range  for  aircraft  from  Kirtland.  The  site  consists  of  open,  semi-arid  terrain.  The  area 
is  relatively  flat,  open  grassland  with  elevation  increasing  from  5100  feet  above  sea  level  on  the 
west,  to  5400  feet  above  sea  level  on  the  east.  Documentation  in  Bureau  of  Indian  Affairs  files 
indicate  that  this  area  was  used  as  a  practice  bombing  range  from  1956  to  1961  to  determine  the 
performance  of  fast  aircraft  during  bombing  runs.  In  the  1960’s,  Kirtland  collected  and  piled 
visible  ordnance  debris  for  removal.  Up  to  2  tons  of  practice  bombs  and  ordnance  waste  per  acre 
were  removed  but  no  explosive  ordnance  was  found. 

JPG,  a  55,265  acre  facility  established  in  December  1940,  fired  its  first  round  5  months  later,  and 
operated  until  1995.  JPG’s  primary  mission  was  to  perform  production  and  post-production  tests 
of  conventional  ammunition  components  and  other  ordnance  items  and  conduct  tests  of 
propellant  ammunition/weapons  systems  and  components  for  the  U.S.  Army.  JPG  is  located  in 
southeastern  Indiana,  approximately  8  miles  north  of  the  Indiana-Kentucky  border  and  about  5 
miles  north  of  Madison,  Indiana.  The  installation  occupies  parts  of  Jefferson,  Jennings  and 
Ripley  Counties,  and  is  about  17.2  miles  in  length  and  ranges  from  4  to  6  miles  in  width.  Lands 
surrounding  JPG  are  predominantly  farmland  and  woodlands,  with  some  small  towns  and  rural 
residential  land  use  nearby. 

The  former  Fort  Ord  is  located  80  miles  south  of  San  Francisco  and  occupies  approximately 
28,000  acres  adjacent  to  Monterey  Bay  and  the  cities  of  Marina,  Seaside,  Sand  City,  Del  Rey 
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Oaks,  and  Monterey.  Fort  Ord  became  a  training  installation  in  1917  and  was  used  to  train  Army 
infantry,  cavalry,  and  field  artillery  divisions  for  WWI  and  II,  Korea,  Vietnam,  and  Desert 
Storm.  The  topography  of  the  Seaside  area  at  the  fonner  Fort  Ord  is  flat  to  gently  rolling  terrain. 
The  vegetation  is  comprised  of  maritime  chaparral  and  oak  woodlands. 


3.4  Present  Operations 

APG  is  currently  home  to  a  diverse  array  of  weapons  development  and  testing  programs.  The 
APG  Standardized  Test  Sites  have  been  -  and  continue  to  be  -  utilized  to  benchmark  a  significant 
number  of  technologies  and  contractors. 

The  former  Pueblo  Precision  Bombing  and  Pattern  Gunnery  Range  #2  are  primarily  Federal 
lands  that  are  managed  by  the  U.S.  Forest  Service  as  the  Comanche  National  Grasslands,  with 
portions  leased  to  private  owners  or  owned  by  the  State  of  Colorado.  There  is  some  private 
ownership  of  parcels  in  the  middle  of  the  study  area.  All  privately  owned  lands  within  the  study 
area  are  used  for  cattle  grazing.  The  general  recreational  use  of  the  site  is  very  broad  and 
encompasses  hiking,  camping,  and  use  by  all-terrain  vehicles.  The  entire  site  is  also  used  for 
cattle  grazing,  which  may  require  well  drilling  and  pipe  laying  to  supply  water  to  the  cattle,  as 
well  as  fences. 

The  Isleta  Pueblo  is  currently  home  to  the  Tigua  tribe.  The  area  of  interest  is  primarily  used  for 
raising  cattle,  horses  and  sheep  by  Isleta  ranchers. 

In  1989,  JPG  was  identified  for  base  closure  under  the  BRAC  program.  The  base  was  closed  in 
1995  and  the  property  disposal  process,  which  included  UXO  removal  actions,  began  the 
following  year.  The  U.S.  Fish  and  Wildlife  Service  entered  into  Memorandums  of  Agreement 
with  the  Anny  and  established  the  Big  Oaks  National  Wildlife  Refuge  to  help  ensure  a  balance 
use  of  the  land  and  maintenance  of  the  natural  resources.  The  Jefferson  Range  air  to  ground 
training  area  is  also  currently  used  by  the  Indiana  Air  National  Guard. 

In  1991,  Fort  Ord  was  included  on  the  BRAC  list  and  closed  in  1994.  Since  the  BRAC  listing 
and  closure  of  Fort  Ord,  cleanup  operations  have  been  performed  to  address  explosive  hazards 
and  to  prepare  Fort  Ord  property  to  be  transferred  to  federal,  state,  and  local  agencies  and  the 
surrounding  Monterey  County  communities. 


3.5  Pre-Demonstration  Testing  and  Analysis 

This  Demonstration  used  target  picking  algorithms  that  have  been  prototyped  and  undergone 
limited  testing  during  previous  research  programs.  Three  of  the  algorithms  (AWD,  Matched 
filter  and  clustering)  have  been  seamlessly  integrated  into  Oasis  montaj™. 
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Prior  to  the  demonstration,  we  verified  that  the  Oasis  version  produced  the  same  results  as  the 
prototype  code.  The  AWD  and  Matched  filter  actually  called  the  same  DLL’s  as  the  prototype. 
Only  the  clustering  algorithm  needed  significant  recoding  in  order  to  run  in  the  Oasis  montaj™ 
environment. 


3.6  Testing  and  Evaluation  Plan 

3.6.1  Demonstration  Set-up  and  Start-up 
Not  applicable  to  this  demonstration. 

3.6.2  Period  of  Operation 

The  schedule  for  the  major  items  in  the  Demonstration  is  given  in  tabular  form  in  Table  3-3. 


Table  4.  Target  Picking  Demonstration  Schedule 


Date 

Action 

November  2006 

Receive  approval  for  Demonstration  Plan. 

December  1-15,  2006 

Phase  1  testing  using  a  60  dipole  synthetic  data  set 

December  15,2006  - 
January  15,  2007 

Applied  analytic  signal  and  matched  filter  algorithms  to  all 
demonstration  data  sets. 

January  15  -  February  1, 
2007  * 

Applied  clustering  algorithm  to  3  of  7  demonstration  data  sets 

January  15  -  February 
15,2007 

Applied  AWD  algorithm  to  all  demonstration  data  sets 

May  2007 

Presented  preliminary  results  at  IPR 

July  2007 

Received  new  TAA  with  Geosoft  and  fixed  memory  related  bug  in 
clustering  algorithm 

July  15  -  August  1,  2007 

Applied  clustering  algorithm  to  final  4  demonstration  data  sets 

March  2008 

Submitted  ESTCP  Draft  Final  Report 

3.6.3  Area  Characterized  or  Remediated 

The  demonstration  was  broken  up  into  two  phases.  The  first  phase  used  the  60  dipole  synthetic 
data  set  described  in  section  2.2  to  explore  the  parameter  space  and  optimize  the  algorithms  for 
the  four  automatic  target  pickers.  The  result  of  phase  one  was  a  set  of  starting  parameters  that 
was  used  in  phase  two.  The  second  phase  applied  the  target  pickers  to  the  seven  magnetic  data 
sets  described  in  section  3.2  using  the  parameters  output  from  phase  one.  Because  each  data  set 
has  its  own  unique  data  characteristics  the  starting  parameters  were  adjusted  iteratively  to 


35 


achieve  the  best  performance.  The  knowledge  gained  from  phase  one  was  used  to  guide  these 
adjustments. 

3.6.4  Residuals  Handling 
Not  applicable  to  this  demonstration. 

3.6.5  Operational  Parameters 

Each  of  the  target  pickers  that  were  evaluated  during  this  demonstration  has  parameters  that  need 
to  be  tuned  to  achieve  optimum  performance.  The  starting  parameters  were  calculated  during 
phase  one  of  the  demonstration.  Even  though  the  different  target  pickers  have  different 
parameters  the  process  of  adjusting  the  starting  parameters  to  achieve  the  best  performance  was 
the  same.  The  process  involved  running  the  algorithm  on  the  data  sets  (for  large  data  sets  a  small 
portion  of  the  data  was  used)  and  changing  the  parameters  based  on  a  visual  review  of  the 
results.  Several  iterations  were  run  until  the  optimal  parameters  were  chosen.  The  visual  review 
involved  looking  at  the  types  of  anomalies  that  were  visible  in  the  data  and  changing  the 
parameters  to  select  the  desired  anomalies.  We  also  looked  at  all  the  automatic  picks  and 
determined  if  they  were  valid  or  in  the  noise  levels  of  the  data  set  and  adjusted  the  threshold 
accordingly. 

Figure  17  and  Figure  18  show  the  parameter  dialog  boxes  for  the  AWD  and  clustering 
algorithms,  respectively.  The  match  dipole  algorithm  iterates  over  six  depths  so  the  only 
parameter  to  set  is  the  size  of  the  filter  box.  Once  the  match  filter  output  and  derived  model 
parameters  are  calculated  they  may  be  combined  using  Boolean  logic  to  minimize  the  processing 
artifacts  in  the  match  filter  output  that  are  caused  when  the  filter  window  contains  only  a  portion 
of  the  dipolar  signal.  The  only  parameters  to  set  for  the  analytic  signal  are  the  number  and  type 
of  filters  to  apply  to  the  analytic  signal  grid  prior  to  running  the  peak  picking  algorithm. 


Figure  17.  Parameter  dialog  box  for  AWD  algorithm. 
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Figure  18.  Parameter  dialog  boxes  for  clustering  algorithm. 


3.6.6  Demobilization 
Not  applicable  to  this  demonstration. 


37 


3.7  Selection  of  Analytical/Testing  Methods 


Not  applicable  to  this  demonstration. 


3.8  Selection  of  Analytical/Testing  Laboratory 

Not  applicable  to  this  demonstration. 
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4.  Performance  Assessment 


4.1  Performance  Criteria 


Table  5.  Performance  Criteria  for  this  Demonstration 


Performance 

Criterion 

Description 

Primary  or 
Secondary 

Ease  of  Use 

Describe  the  processing  flow  and  the  anticipated  skill  level 
required.  The  goal  of  the  program  is  that  data  processors 
experienced  with  data  analysis  and  Oasis  montaj™  could 
utilize  the  software  by  reading  a  manual  but  without  hands- 
on  training. 

Primary 

Robustness 

No  major  bugs  that  artificially  limit  the  analysts’  ability  to 
conduct  analysis. 

Primary 

Probability  of 
Detection  (Pd) 
ground  truth 

(#  of  detections  picked  by  automatic  picker  that  are 
coincident  to  ground  truth  target)/(#  of  ground  truth 
targets) 

Primary 

Probability  of 
Detection  (Pd) 
manual 

(#  of  detections  picked  by  automatic  picker  that  are 
coincident  to  a  manual  pick)/(#  of  detections  picked 
manually  by  expert) 

Primary 

Background  Alarm 
Rate  (rBAR) 

(#  of  detections  picked  by  automatic  picker  but  not 
matching  ground  truth  object)/  (arbitrary  number) 

Primary 

Probability  of 

False  Alarm  (Pfa) 
manual 

(#  of  detections  picked  by  automatic  picker  but  not  picked 
by  expert)/  (#  of  detections  picked  manually  by  expert) 

Primary 

Setup  Time 

Time  required  to  setup  optimum  parameters 

Secondary 

Processing  Time 

(Time  required  to  run  automatic  picker)/(time  for 
commercial  expert  to  pick  anomalies) 

Primary 

Detection  Location 
Accuracy 

The  average  horizontal  distance  between  picked  and 
ground  truth  location 

Primary 

Characterization 

Size  Accuracy 

(The  absolute  difference  between  the  fitted  target  diameter 
and  the  ground  truth  diameter)/(ground  truth  diameter) 

Secondary 

Characterization 
Location  Accuracy 

The  average  horizontal  distance  between  the  fitted  XY 
position  and  ground  truth  location. 

Secondary 

Characterization 
Depth  Accuracy 

The  average  difference  between  the  fitted  depth  and 
ground  truth  depth. 

Secondary 

For  the  baseline  manual  picking  demonstration,  the  objective  for  both  the  production  and  WAA 
surveys  was  to  identify  all  anomalies  that  warrant  further  investigation  and  analysis.  In  the 
analysis  for  Standardized  Test  Site  demonstrations,  this  is  referred  to  as  the  Response  Stage.  The 
person  performing  this  target  selection  was  an  experienced  analyst.  The  analyst  was  asked  to 
decide  on  the  threshold  of  the  given  data  set,  document  the  rationale  behind  the  choice,  and  then 
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select  all  anomalies  above  this  threshold.  The  threshold  was  based  on  the  overall  noise  levels  of 
the  data  set  and  was  different  for  each  data  set.  It  was  set  at  a  point  just  above  the  background 
noise  to  maximize  the  number  of  detections  while  minimizing  false  alarms.  The  analyst 
estimated  the  location  of  each  anomaly,  using  his  preferred  analysis  environment.  The  analyst 
did  not  have  a  method  to  estimate  the  spatial  footprint  of  each  anomaly,  so  SAIC  made  these 
estimates  using  the  analyst’s  dig  sheet  and  data. 

All  the  production  survey  data  sets  compared  the  declarations  made  by  the  different  methods  to 
the  available  ground  truth  to  calculate  the  Pd.  All  the  WAA  survey  data  sets  compared  the 
declarations  made  by  the  automatic  pickers  to  the  picks  made  by  the  commercial  expert  to 
calculate  the  Pd  and  Pfa  because  ground  truth  did  not  exist.  The  maximum  Pd  is  1  but  the 
maximum  Pfa  could  be  greater  than  1  because  it  is  based  on  the  number  of  manual  picks.  A  high 
Pfa  means  the  automatic  picker  selected  a  lot  of  anomalies  that  were  not  picked  by  the  manual 
method.  In  general  a  high  Pfa  is  bad  because  it  most  likely  means  that  the  automatic  pickers 
started  picking  noise  in  the  data.  To  verify  this  we  analyzed  the  false  alarms  to  see  if  they  were 
due  to  picking  noise  in  the  data  or  caused  by  valid  targets  that  the  manual  method  missed. 

The  ground  truth  for  APG  is  well  documented.  For  this  reason  the  rBAR  and  location  accuracy 
were  also  calculated  for  the  APG  data  set.  Because  only  a  select  number  of  targets  have  been 
released  to  the  public  the  scoring  for  the  APG  data  set  will  be  done  by  the  program  office. 

This  demonstration  resulted  in  32  sets  of  scores.  A  set  of  scores  for  each  target  picking  method 
(5)  run  on  each  of  the  production  survey  data  sets  gave  25  scores.  The  WAA  airborne  data  was 
compared  to  the  manual  picks  and  produced  4  sets  of  scores.  Finally,  the  WAA  transect  data 
consisted  of  only  3  sets  of  scores  because  the  matched  filter  algorithm  was  not  be  applied  and  the 
other  automatic  methods  were  compared  to  the  manual  picks. 


4.2  Performance  Confirmation  Methods 

The  ‘Ease  of  Use’  criterion  reports  the  analysts’  experience  regarding  the  time  and  experience 
required  for  a  user  to  leam  to  use  each  automatic  target  picker  and  set  up  optimum  parameters  as 
well  as  the  level  of  user  interaction  required.  It  will  be  derived  from  notes  taken  by  the  analysts. 

The  ‘Robustness’  criterion  reports  the  analysts’  experience  relating  to  bugs  and  program 
deficiencies  that  hinder  efficient  data  analysis.  The  Robustness  report  will  be  derived  from  notes 
taken  by  the  analysts. 

The  Probability  of  detection,  background  alarm  rate,  and  location  and  depth-accuracy  criterions 
will  be  calculated  from  the  results  of  our  analysis  and  the  provided  ground  truth  infonnation. 

Analysis  time  will  be  logged  manually.  To  provide  meaningful  baseline  metrics,  we  will 
separately  record  time  spent  A)  selecting  the  optimum  parameters  for  each  target  picker,  and  B) 
computational  time  to  execute  the  target  picker. 
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A  successful  demonstration  would  be  defined  as  having  some  or  all  methods  meeting  the  primary 
performance  criteria  previously  outlined  in  Table  1,  Table  2  and  Table  5  for  some  or  all  of  the 
data  sets.  Areas  with  high  target  density  where  ambiguous  results  impeded  our  analysis  were 
removed  from  the  ground  truth  comparison.  We  conducted  a  failure  analysis  on  the  data  sets  that 
did  not  meet  the  performance  criteria.  The  failure  analysis  detennined  if  data  quality  was  the 
determining  factor  in  not  reaching  the  performance  criteria  or  if  the  problem  resided  in  the  target 
picking  algorithm  or  other  factors.  If  the  failures  were  due  to  data  quality  the  demonstration  was 
still  deemed  successful  and  these  limitations  were  noted  in  the  final  report.  It  is  not  fair  to 
penalize  the  target  picking  algorithms  for  poor  data  quality. 


4.3  Data  Analysis,  Interpretation,  and  Evaluation 

4.3.1  Phase  1  -  Synthetic  Magnetic  data 

The  first  phase  of  the  demonstration  used  the  60  dipole  synthetic  data  set  described  in  section  2.2 
to  explore  the  parameter  space  and  optimize  the  algorithms  for  the  four  target  pickers.  We 
varied  their  different  parameters  to  evaluate  and  document  their  effect  and  to  determine  if  certain 
“rules  of  thumb”  could  be  developed.  For  example,  changing  certain  parameters  may  allow 
deeper  or  larger  targets  to  be  selected.  This  infonnation  was  then  used  to  guide  us  when 
applying  the  target  pickers  to  the  data  sets  in  the  demonstration.  In  addition  to  this  knowledge, 
phase  one  was  also  used  to  select  a  set  of  starting  parameters  for  each  automatic  picker  that  was 
used  in  phase  two.  The  perfonnance  of  the  different  automatic  target  pickers  is  summarized  in 
Figure  19.  As  we  can  see  none  of  the  automatic  target  pickers  detected  all  the  targets.  The 
analytic  signal  and  the  wavelet  produced  the  best  results  followed  by  the  clustering  algorithm 
and  finally  the  matched  filter.  All  60  targets  cannot  be  recovered  by  any  of  the  target  pickers 
without  including  thousands  of  false  alarms  because  the  amplitudes  of  the  missed  targets  are  of 
an  amplitude  comparable  to  that  of  the  background  noise.  The  SNR  for  the  1 1  targets  that  were 
consistently  missed  by  all  the  target  pickers  ranged  from  -1.7  to  9.2.  The  SNR  is  defined  as 
20xlog(peak  signal/noisc)  where  the  noise  is  the  standard  deviation  of  the  synthetic  noise  within 
a  3  meter  circle  surrounding  target  location.  In  general  the  smaller  20mm  and  40mm  targets 
were  missed. 
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Figure  19.  Receiver  Operating  Characteristic  (ROC)  curve  of  the  different  target  pickers  for  the 
synthetic  60  dipole  data.  The  symbols  along  each  line  represent  the  analyst’s  threshold  value. 
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4.3.1 . 1  Automated  Wavelet  Detection  algorithm 


Target  picking  with  the  AWD  algorithm  is  a  two  stage  process:  the  first  step  is  the  detection  of 
anomalies  above  a  defined  threshold;  the  second  is  the  Upward  Continuation  (UpC)  of  these 
anomalies  to  determine  their  persistence  and  convergence  to  an  original  dipole.  Performance  of 
AWD  is  controlled  by  a  set  of  parameters  that  the  user  can  adjust  to  the  characteristics  of  a  data 
set.  During  our  testing  on  the  synthetic  data  we  have  found  the  following  generalizations  for  the 
different  parameters. 

The  two  parameters  that  have  the  largest  effect  on  the  number  of  detected  targets  are  the 
"minimum  amplitude  of  dipole"  and  the  "minimum  amplitude  of  a  single  peak".  These 
parameters  define  the  amplitude  threshold  for  an  anomaly  to  be  considered  as  a  potential  target. 
The  values  for  these  amplitudes  should  be  set  by  the  user  to  a  value  above  the  estimated  noise  in 
the  dataset. 

The  AWD  algorithm  searches  for  the  negative  peak  associated  to  a  positive  in  a  circle  of  radius 
defined  by  the  parameter  "maximum  distance  between  peaks",  which  can  be  increased  when 
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AWD  obviously  fails  to  associate  anomalies  for  the  same  target.  For  a  dipole  or  “monopole” 
(actually  a  vertical  or  near-vertical  dipole)  anomaly  to  be  identified  as  a  target,  the  peaks  of  the 
anomaly  must  persist  at  several  levels  through  UpC.  Furthermore,  the  origin  of  these  peaks  must 
converge  to  dipoles  within  a  given  depth  and  distance  range  and  the  peaks  must  move  together 
within  a  given  angular  tolerance. 

The  parameter  "start  height"  defines  the  first  level  at  which  UpC  should  be  started.  We  have 
found  that  it  is  best  to  keep  that  parameter  to  0  if  the  noise  level  in  the  data  is  not  too  high  in 
order  to  maximize  the  probability  of  detection;  otherwise  the  parameter  can  be  increased  to  a 
survey  pixel  size  (e.g.,  0.125  m)  to  filter  out  small  short  wavelength  noise. 

The  parameter  "end  height"  generally  requires  no  change  unless  one  is  particularly  interested  in 
large  deep  targets,  whose  signal  would  persist  at  high  UpC  levels,  or  to  increase  the  potential  of 
detection. 

The  "height  increment"  for  the  levels  of  UpC  should  be  set  to  a  pixel  size,  or  modified  to 
increase/decrease  the  chance  of  an  anomaly  to  be  kept  as  a  target.  This  parameter  works  in 
combination  with  the  "minimum  persistence",  which  defines  the  number  of  levels  through  which 
an  anomaly  must  persist  to  be  identified  as  a  target.  The  latter  can  be  increased  when  data  are 
contaminated  with  short  wavelength  noise,  e.g.,  as  for  the  synthetic  data  set. 

The  "lowest  level"  can  also  be  increased/  decreased  to  include/exclude  potentially  shallow 
targets  with  small  amplitude. 

Selection  of  deep  targets  is  set  by  the  "maximum  depths"  parameter. 

Positive  and  negative  peaks  are  associated  if  they  have  the  same  estimated  source.  In  practice, 
noise,  interference  and  limited  resolution  of  the  data  can  limit  convergence.  The  algorithm 
requires  instead  that  sources  converge  within  a  given  depth  (parameter  "maximum  difference  in 
peak  amplitude",  set  to  0.25,  can  be  lowered  to  0.2)  and  distance  range  (parameter  "maximum 
distance  away  as  a  function  of  depth",  set  to  1.5,  can  be  increased  to  2).  Peaks  must  also  move 
toward  one  another  across  scales  within  a  given  angular  sector  (parameter  "maximum  mismatch 
in  angles",  set  to  90  degrees,  up  to  120  degrees).  These  parameters  can  be  altered  when 
changing  the  "maximum  distance  between  peaks"  is  ineffective  at  helping  AWD  associate 
anomalies  for  the  same  target. 

Using  the  above  knowledge  45  of  the  60  targets  can  be  picked  without  any  false  alarms,  which  is 
an  improvement  over  the  results  from  the  shakedown  described  in  section  2.2.1.  Additional 
targets  can  be  detected  at  the  cost  of  including  false  alarms:  48  targets  are  found  with  2  errors 
(Figure  20),  49  for  18.  Among  the  key  parameters,  the  minimum  amplitude  of  the  dipoles  and 
single  poles  are  the  most  important  ones.  For  this  synthetic  dataset  the  threshold  has  to  be  set 
quite  high  at  a  minimum  of  30nT  to  limit  picking  of  noise  artifacts.  Up  to  48  targets  can  be 
picked  in  that  manner,  depending  on  the  value  of  the  other  parameters.  Parameters  for  the  best 
results  are  presented  in  Table  6(for  pixel  size  of  0. 125  m). 
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Additional  targets  were  picked  up  by  lowering  the  amplitude  thresholds.  For  instance,  49  targets 
were  obtained  with  minampdipole  =  25  nT  and  minampsingle  =  20  nT,  with  all  other  parameters 
being  the  same.  False  alarms  can  be  weaned  out  by  setting  a  high  value  to  the  minimum 
persistence,  e.g.,  7  when  the  default  value  is  2. 
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Figure  20.  Results  of  AWD  algorithm  on  the  synthetic  magnetic  data.  The  true  locations  are 
plotted  as  circles  and  the  automatic  picker  detections  are  plotted  as  “+” 


4. 3. 1.2  Clustering  algorithm 

The  clustering  parameters  can  be  divided  into  three  main  groups:  target  detection,  data  extraction 
and  target  overlap  detection.  The  main  parameters  that  affect  target  detection  are  the  amplitude 
threshold,  search  radius,  and  dipole  stretch.  During  our  testing  the  first  two  behaved  as  expected. 
Either  increasing  the  amplitude  threshold  or  increasing  the  search  radius  decreases  both  the 
number  of  true  targets  found  and  the  number  of  spurious  targets.  The  latter  tend  to  fall  off  faster 
at  low  values.  We  found  that  a  good  amplitude  threshold  was  approximately  3  times  the  noise 
range  of  the  data.  We  estimated  the  noise  range  by  subtracting  the  amplitude  at  the  69th 
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percentile  from  the  amplitude  at  the  3 1st  percentile  of  the  cumulative  distribution  function  (CDF). 
These  values  are  automatically  calculated  and  displayed  in  the  input  parameters  dialog  box  for 
the  user  to  either  accept  or  override.  The  values  can  be  overridden  by  either  altering  the 
percentiles  used  or  by  inputting  the  actual  amplitude  threshold  to  be  used.  The  search  radius 
should  be  set  to  at  least  as  large  as  the  cross  track  spacing  of  the  survey  data;  otherwise,  the 
algorithm  may  be  calling  something  a  peak  by  only  looking  up  and  down  track.  Ideally,  the 
search  radius  should  also  be  less  than  the  separation  between  adjacent  dipoles  because  only  the 
largest  dipole  within  the  search  radius  is  flagged  as  a  target.  Unfortunately,  these  two  criteria 
may  be  exclusive.  If  so,  the  user  must  choose  which  one  to  apply.  The  dipole  stretch  should  be 
large  enough  to  cover  the  widest  observed  lobe  separation.  If  the  dipole  stretch  is  set  too  small 
the  positive  and  negative  lobes  of  certain  large  targets  may  be  not  get  associated  and  will  be 
counted  as  separate  targets.  On  the  other  hand,  a  dipole  stretch  that  is  too  large  may  increase  the 
number  of  possible  lobes  of  opposite  sign  that  is  associated  with  each  peak.  This  in  turn  will 
increase  the  possibility  of  associating  the  wrong  two  lobes.  Also  with  too  large  a  dipole  stretch, 
single  lobes  will  often  find  an  erroneous  secondary  lobe  that  is  too  far  away  in  the  secondary 
association.  This  results  in  errors  for  the  final  calculated  location  for  those  dipoles. 

The  parameters  that  mainly  affect  data  extraction  are  the  footprint  (maximum  radius),  the  noise 
range,  and  the  number  of  slices.  The  footprint  should  be  large  enough  to  cover  the  largest  visible 
dipole.  The  noise  range  as  describe  above  is  a  good  starting  point.  It  is  used  both  as  a  stopping 
criterion  in  moving  out  from  the  peaks,  and  in  the  second  association.  The  data  within  a  circle 
around  each  peak  is  divided  into  slices.  The  algorithm  sorts  all  the  data  within  each  slice  by 
distance  to  the  peak.  For  each  slice  the  algorithm  looks  down  the  sorted  data  for  a  gradient 
crossover  to  define  the  stopping  criterion  for  the  cluster.  During  our  extensive  testing  we  found 
that  using  24  slices  gives  good  results  and  should  not  need  to  be  changed. 

The  parameters  that  control  the  overlap  detection  are  ratio  threshold,  slice  angle  expansion  and 
overlap  distance.  These  parameters  only  detennine  if  the  overlap  flag  is  set  and  do  not  affect  the 
number  of  targets  detected  or  their  association.  For  this  reason  they  are  not  as  important  and  the 
default  settings  are  normally  satisfactory. 

The  clustering  algorithm  found  46  of  the  60  targets  with  1 1  false  positives  using  a  34nT 
amplitude  threshold  as  shown  in  Figure  21.  Among  the  parameters,  the  amplitude  threshold  and 
search  radius  were  the  most  important  for  this  data  set.  One  target  (2nd  row  from  bottom  and  4th 
column)  had  a  pick  that  was  just  outside  the  1  meter  tolerance.  The  positive  peak  of  this  dipole 
was  correctly  selected  but  was  associated  with  the  wrong  negative  peak  causing  the  final  location 
error. 


45 


328040 


0  33 

20™  60rrim 


328040 


328060 


328060 


328000  328020 


328000 


105mm 

058 
$  105mn 
0.J5  f 

.  60mm 

40mm  Q.  * 

^  0  46  0.12  I 


ISSi 

'.40  0.M 

20mm  H"  ^ 

o  C  « 

045  00|  080 

20mm  40mrf  81mm 

'  "  &  * 

aZ°  4-tefj  0-27 

155mm  O  i  20™1 

$  0.31  '  O 

v1a1Q  0.11 

*T  60mm 

&■  A  > — #  _  ,  _ 

0  04  'f,  0  20  1.0#  0.07  0.45 

105mm  0.07  |  40mm  155ntti  ,«_-l 

0  50  V'  0  22  1.53  A  0f6 

;4r  "*  20mm 

^  ~  0.20 


105mm  i0"™ 


0  35  0  81  0  ,3 

105mm  20mm  81mm 


_ flJjmm.  Q  . 


0.02 


60mm  20mm 

$  o 


155mm 
its  155mm 

o“  * 


■  0  62 


1  53 


— N 


Synthetic  Magnetic  Data  ooscnpcran 

Total  Field  Magnetics  O  Seeded  target 

a  Clustering  Algorithm 

34nT  threshold  +•  Autopicked  target 


Figure  21.  Results  of  clustering  algorithm  on  the  synthetic  magnetic  data.  The  true  locations  are 
plotted  as  circles  and  the  automatic  picker  detections  are  plotted  as 


4.3. 1.3  Analytic  signal 


The  analytic  signal  method  only  has  two  parameters  to  change:  1)  the  number  of  filters  to  apply 
to  the  analytic  signal  grid  and  2)  the  amplitude  threshold  for  the  peak  picking  algorithm.  To  test 
these  parameters,  the  Geosoft  peak  detection  routine  was  run  using  the  analytic  signal  grid  with  0 
to  5  passes  of  a  3x3  Hanning  filter  applied.  The  amplitude  threshold  for  the  peak  picking  routine 
was  set  to  value  sufficiently  small  to  pick  at  least  1000  targets.  The  actual  value  of  the  amplitude 
threshold  was  different  with  each  grid  because  the  grid  amplitudes  decreased  with  each 
successive  pass  of  the  Hanning  filter.  The  picks  were  sorted  from  largest  to  smallest  based  on 
the  analytic  signal  amplitude  and  the  results  are  shown  in  Figure  22.  It  is  clearly  seen  that  at 
least  one  pass  of  the  filter  is  needed  to  get  decent  results.  In  this  case,  two  passes  of  the  filter 
shows  a  definite  improvement  over  one  pass.  There  does  not  appear  to  be  much  advantage  in 
applying  three  or  more  filter  passes. 
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ROC  curve  using  Analytic  Signal  with  various 
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Figure  22.  .ROC  curve  of  the  analytic  signal  algorithm  applied  to  the  60  dipole  synthetic  data  set 
using  different  number  of  3x3  Hanning  filters. 

As  stated  in  Section  2.2.4,  the  analytic  signal  algorithm  found  49  of  the  60  targets  with  no  false 
positives  using  a  45nT/m  amplitude  threshold.  Two  passes  of  a  3x3  Hanning  filter  were  applied 
to  the  analytic  signal  grid.  The  targets  missed  consisted  of  nine  20mm  at  various  depths  and  two 
40mm  that  were  buried  at  0.4  meters. 

4.3.1 .4  Matched  Filter  Algorithm 

The  matched  filter  algorithm  has  only  one  main  parameter  (filter  box  size)  to  set  but  because  it 
runs  on  the  gridded  magnetic  data  care  must  be  taken  when  generating  the  input  grid.  We  found 
that  a  good  starting  grid  cell  for  the  matched  filter  input  grid  is  1/4  of  the  mean  track  spacing. 
Increasing  the  grid  cell  size  will  decrease  the  processing  time  because  the  grid  will  contain  fewer 
grid  cells.  But  if  the  cell  size  is  set  too  large,  smaller  anomalies  begin  to  disappear  from  the  grid. 
Data  gaps  in  the  input  grid  create  problems  for  the  matched  filter  algorithm  because  they  are 
enlarged  by  one-half  the  size  of  the  filter  box  window  during  the  matched  filtering  process.  As  a 
result,  anomalies  near  the  margin  of  data  gaps  will  be  missed  if  the  data  gaps  are  not  interpolated 
during  the  gridding  process.  By  allowing  data  gaps  to  be  interpolated  during  the  gridding 
process,  the  problems  associated  with  data  gaps  are  minimized.  Oasis  montaj™  allows  the  user 
to  define  the  distance  over  which  grid  values  are  interpolated.  The  built  in  Oasis  montaj™ 
interpolation  routines  are  based  on  published  schemes  (Briggs,  1974  and  Swain,  1976). 

A  filter  box  is  created  around  each  point  in  the  magnetic  grid  and  the  data  within  the  filter  box  is 
convolved  using  a  dipole  based  model.  The  output  of  the  matched  filter  algorithm  is  maximized 
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at  locations  that  show  a  good  approximation  to  the  dipole  model.  The  filter  box  size  should  be 
set  large  enough  to  cover  the  largest  dipole  but  not  too  large  because  the  run  time  scales  as  the 
filter  box  size  squared.  So  using  a  slightly  smaller  filter  box  that  does  not  degrade  the  results 
will  greatly  decrease  the  run  time.  Adjusting  the  filter  box  in  conjunction  with  the  input  grid  cell 
size  described  above  will  improve  the  overall  run  time  of  the  algorithm. 

The  matched  filter  output  data,  shown  in  section  2.2.3,  clearly  indicated  the  presence  of  some  of 
the  60  dipoles  but  processing  artifacts  are  also  apparent.  These  artifacts  are  evidenced  by  the 
numerous  false  alarms  and  the  multiple  picks  surrounding  some  of  the  larger  anomalies.  These 
additional  picks  occurred  even  though  the  output  grid  was  filtered  several  times  to  smooth  out 
the  errors.  The  artifacts  occurred  when  the  filter  window  contained  only  a  portion  of  the  dipolar 
signal  (typically  a  single  positive  or  negative  lobe)  that  is  associated  with  the  causative  source. 
When  this  happened,  the  filter  incorrectly  created  the  missing  lobe,  which  produced  erroneous 
output  values.  The  matched  filter  algorithm  stored  the  output  filter  results  and  all  derived  model 
parameters  in  a  database  for  each  grid  cell.  These  model  parameters  were  used  to  refine  the  filter 
output.  The  matched  filter  output  was  divided  by  the  Chi  square  model  error  (‘Chisq’)  to 
produce  the  ratio  (‘MFoutput/Chisq’)  shown  in  Figure  23.  When  the  filter  window  coincides 
with  the  center  of  a  dipolar  signal,  the  error  tenn  is  dramatically  reduced.  In  other  words,  the 
derived  model  closely  matches  the  input  (measured  or  synthetic)  data  when  it  is  directly  on  top 
of  the  center  of  the  dipole.  As  observed  in  the  figure,  the  anomalies  now  possess  a  very  high 
signal-to-noise  ratio  and  the  processing  artifacts  are  reduced. 

The  MFoutput/Chisq  grid  was  input  to  Geosoft’s  peak  detection  routine.  As  with  the  analytic 
signal,  the  input  grid  needs  to  be  filtered  two  or  three  times  with  a  3x3  Hanning  filter  to  produce 
the  best  results.  A  very  small  threshold  was  used  and  the  results  were  sorted  from  largest  to 
smallest  ratio  amplitude.  The  results  were  analyzed  to  locate  the  threshold  value  that  maximized 
the  targets  detected  while  minimizing  the  false  alarms.  Using  a  threshold  value  of  2.1  the 
matched  filter  method  detected  43  targets  with  17  false  alarms.  Compared  to  the  results  of  using 
only  the  matched  filter  output  presented  in  section  2.2.3,  the  ratio  found  the  same  number  of 
targets  with  29  fewer  false  alarms. 
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Figure  23.  Results  of  matched  filter  algorithm  on  the  synthetic  magnetic  data.  The  color  coded 
map  shows  the  ratio  (Matched  filter  output/Chi  square).  The  true  locations  are  plotted  as  circles 
and  the  automatic  picker  detections  are  plotted  as  “+” 


4.3.2  Phase  2  -  Live  Site  data 

The  second  phase  applied  the  target  pickers  to  the  seven  magnetic  data  sets  using  the  default 
parameters  output  from  phase  one.  Because  each  data  set  had  its  own  unique  data  characteristics 
the  starting  parameters  needed  to  be  adjusted  to  achieve  the  best  performance.  The  knowledge 
gained  from  phase  one  was  used  to  guide  these  adjustments.  Even  though  the  different  target 
pickers  had  different  parameters  the  process  of  adjusting  the  starting  parameters  to  achieve  the 
best  performance  was  essentially  the  same.  The  process  involved  running  the  algorithm  on  the 
data  sets  (for  large  data  sets  a  small  portion  of  the  data  was  used  for  time  efficiency)  and 
changing  the  parameters  based  on  a  visual  review  of  the  results.  This  was  an  iterative  process  of 
intelligently  changing  parameters  and  visually  looking  at  the  results.  The  visual  review 
examined  the  different  types  of  anomalies  that  were  selected  and  those  that  were  not  selected  and 
altering  the  parameters  accordingly.  One  of  the  limitations  noted  in  past  studies  of  the  automatic 
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pickers  is  their  inability  to  pick  targets  at  or  below  the  local  geologic  signal  without  picking 
numerous  false  targets.  For  this  reason  we  also  looked  at  all  the  automatic  picks  and  detennined 
if  they  are  visually  valid  or  in  the  noise  levels  of  the  data  set  and  adjusted  the  threshold 
accordingly.  By  testing  different  thresholds,  we  were  able  to  determine  the  point  at  which  we 
maximized  the  number  of  targets  picked  while  minimized  the  number  of  apparent  false  alarms. 
Table  6  to  Table  9  summarizes  the  final  parameters  used  on  each  data  set  for  each  of  the 
automatic  target  picking  methods. 

We  calculated  the  Pd  by  comparing  the  declarations  made  by  the  different  picking  methods  to 
the  available  ground  truth  data.  We  eliminated  high  density  areas  where  ambiguous  results 
impeded  our  analysis.  The  WAA  data  sets  did  not  have  sufficient  ground  truth  over  the  areas 
analyzed.  For  this  reason  we  compared  the  automatic  picks  to  the  picks  made  by  the  commercial 
expert.  For  the  APG  data  we  also  calculated  the  rBAR  because  it  was  the  only  data  set  that  had 
complete  ground  truth.  The  three  USACE  data  sets  had  ground  truth  but  only  for  anomalies  that 
were  declared  by  the  contractor.  There  were  anomalies  observed  in  the  magnetic  data  that  were 
not  excavated  so  we  could  not  ascertain  whether  they  were  caused  by  noise  or  a  metallic  object. 
Also,  the  locations  for  the  ground  truth  (except  for  the  proveout)  were  locations  that  were 
selected  by  the  analyst  and  not  the  actual  location  of  the  found  item. 

As  part  of  our  analysis  we  plotted  the  ground  truth  sorted  by  decreasing  analytic  signal  amplitude 
versus  the  number  of  targets  detected  by  each  picking  method.  This  will  ideally  give  a  straight 
line  along  the  diagonal  if  all  the  ground  truth  were  detected.  By  sorting  according  to  signal 
strength  we  are  able  to  visually  see  if  there  are  trends  in  the  types  of  targets  missed  for  the 
different  data  sets.  It  also  allowed  us  to  visualize  the  picking  thresholds  for  the  different 
methods.  We  also  created  maps  (shown  in  the  next  section)  for  each  data  set  and  each  method 
and  overlaid  the  automatic  target  declarations,  ground  truth  found  and  ground  truth  not  detected 
as  well  as  the  manual  declarations. 

Since  characterization  is  a  vital  part  of  any  UXO  cleanup  operation,  the  ability  of  an  automatic 
target  picker  to  define,  for  each  identified  anomaly,  an  appropriate  and  effective  spatial  footprint 
to  be  given  to  a  fitting  (or  characterization)  routine  is  an  important  metric  to  measure  and 
compare  among  the  methods.  We  used  the  spatial  footprint  produced  by  each  method  as  input  to 
a  target  fitting  routine  for  80  anomalies  that  were  picked  by  all  the  methods  over  the  APG  data 
set.  The  effect  that  the  selection  has  on  the  fitted  model  parameters  was  compared  and  included 
in  the  demonstration. 
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Table  6.  Best  Parameter  Set  for  AWD  Algorithm  for  All  Surveys. 


Dataset/ 

Parameter 

Default 

GPO 

Pueblo 

Airb. 

Isleta 

Seaside 

APG 

JPG 

Pueblo 

Trans. 

minampdip 

10 

10 

5 

10 

6 

8 

6 

7 

minampsing 

10 

10 

7 

10 

6 

8 

6 

7 

startheight 

0 

0 

0 

0.1 

0 

0 

0 

0 

endheight 

1.5 

2 

4 

1.6 

1.5 

1.5 

1.5 

1.5 

incheight 

0.125 

0.125 

0.2 

0.125 

0.125 

0.125 

0.125 

0.125 

lowestlev 

4 

3 

10 

4 

3 

2 

2 

3 

minpersist 

2 

2 

4 

3 

2 

2 

2 

2 

maxdepth 

5 

5 

10 

5 

5 

8 

7 

8 

maxangle 

90 

120 

90 

90 

90 

90 

130 

90 

maxdist 

3 

5 

8 

3 

5 

5 

5 

5 

maxampdiff 

0.25 

0.2 

0.25 

0.2 

0.2 

0.25 

0.25 

0.25 

maxdistdep 

1.5 

2.5 

1.5 

2 

2 

1.5 

1.5 

1.5 

Table  7.  Best  Parameter  Set  for  Clustering  Algorithm  for  All  Surveys. 

Dataset/  Parameter  Default  GPO  Pueblo  Isleta  Seaside  APG  JPG  Pueblo 


Airb. 

Trans. 

Amplitude  threshold  (nT) 

3  x 
noise 

175 

4 

30 

9 

15 

7 

10 

Dipole  stretch  (m) 

3 

2 

7.5 

3 

2 

1 

2 

1.75 

Search  radius  (m) 

3 

1 

8 

1.25 

1.5 

3 

1.25 

1.5 

Footprint  radius  (m) 

5 

5 

10 

5 

5 

5 

5 

5 

Overlap  distance  (m) 

2 

2 

7 

2 

2 

2 

2 

2 

Ratio  threshold 

.1 

.1 

.1 

.1 

.1 

.1 

.1 

.1 

Azimuth  deviation  (°) 

55 

179 

179 

179 

179 

179 

179 

179 

Slice  angle  expansion (°) 

25 

25 

25 

25 

25 

25 

25 

25 

Number  of  slices 

24 

24 

24 

24 

24 

24 

24 

24 
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Table  8.  Best  Parameter  Set  for  Matched  Filter  Algorithm  for  All  Surveys. 


Dataset/  Parameter 

Default 

GPO 

Pueblo 

Airb. 

Isleta 

Seaside 

APG 

JPG 

Input  grid  cell  (m) 

%  line 
spacing 

.15 

.9 

.2 

.15 

.2 

.2 

Filter  box  size  (in) 

3 

6 

12 

5.5 

3.75 

5 

4 

Peak  picker  grid  cell  ( m ) 

%  input 
grid  cell 

.075 

.45 

.05 

.0375 

.1 

.05 

Number  filter  passes 

3 

4 

3 

3 

3 

3 

4 

Peak  Threshold 

1 

.3 

.9 

.55 

1.25 

.35 

.7 

Line  spacing  (m) 

NA 

.6 

1.75 

.25 

.6 

.25 

.75 

Table  9.  Best  Parameter  Set  for  Analytic  Signal  Method  for  All  Surveys. 


Dataset/  Parameter 

Default 

GPO 

Pueblo 

Airb. 

Isleta 

Seaside 

APG 

JPG 

Pueblo 

Trans. 

Amplitude  threshold 
( nT/ni ) 

NA 

83.9 

3 

50 

5 

42 

5 

25 

Number  filter  passes 

2 

2 

2 

2 

2 

2 

4 

2 

4.3.2. 1  USACE  GPO 

The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  10  and  Figure 
24.  Color  coded  maps  showing  the  different  target  picking  declarations  and  ground  truth 
overlain  on  their  respective  data  are  presented  in  Figure  25  to  Figure  29.  The  red  symbols  on  the 
figures  represent  items  that  were  not  detected  by  the  respective  method.  The  Pd’s  for  the 
different  methods  were  lower  than  expected  with  only  the  manual  method  approaching  our  goal 
of  0.90.  The  lower  Pd’s  can  be  attributed  to  the  following  factors.  First,  there  were  a  limited 
number  of  seeded  items  so  each  missing  item  significantly  affected  the  Pd.  The  seeded  items 
consisted  of  18  ordnance  items  (171b  and  251b  bombs)  and  10  clutter  items.  Two  of  the  clutter 
items  were  masked  by  a  large  anomaly  that  bisected  the  area  and  were  not  seen  by  any  of  the 
methods.  So  in  order  to  meet  our  goal  all  the  remaining  items  needed  to  be  detected.  Second, 
the  area  had  a  high  magnetic  background  which  forced  higher  thresholds  for  each  of  the 
methods.  This  resulted  in  missing  some  of  the  weaker  clutter  items.  It  should  be  noted  that  both 
the  manual  method  and  the  analytic  signal  method  detected  all  the  emplaced  ordnance  and  the 
clustering  and  AWD  methods  only  missed  two  ordnance  items.  The  vast  majority  of  the  missed 
items  that  led  to  the  lower  Pd’s  were  range  clutter. 

The  matched  filter  method  was  severely  hampered  by  the  requirement  of  the  filter  box  to  have 
complete  data  coverage.  This  effectively  reduced  the  size  of  the  area  that  could  be  analyzed 
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which  in  turn  eliminated  10  of  the  seeded  items.  Of  the  remaining  18  items,  two  ordnance  items 
were  missed.  They  were  both  partially  masked  by  a  large  anomaly,  although  one  of  the  items 
could  have  been  detected  with  a  lower  threshold.  Even  though  these  anomalies  were  visually 
seen  in  the  magnetic  data,  the  nearby  larger  anomalies  created  a  large  Chisq,  which  resulted  in  a 
small  value  for  the  ratio  (MFoutput/Chisq)  and  thus  the  missed  detection. 


Tal 

ble  10.  USACE  GPO 

-  Detection  and  location  accuracy 

Parameter  / 
Method 

Total  # 
picks 

#  matching 
ground 
truth 

Pd 

# 

<.2  5m 

#  .25  m 
-  .5m 

#.5- 

1.0m 

% 

within 

.5m 

Ground  Truth 

28 

Manual 

36 

25 

0.89 

8 

14 

3 

88 

Clustering 

27 

16 

0.57 

5 

5 

6 

63 

Wavelet 

39 

21 

0.75 

9 

8 

4 

81 

Matched  filter 

14 

10 

0.36 

2 

5 

3 

70 

Analytic  signal 

50 

22 

0.79 

5 

16 

1 

95 

Figure  24.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  USAGE  GPO. 
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Figure  25.  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  analytic  signal  method;  USACE  GPO. 
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Figure  26.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  clustering  method;  USACE  GPO. 
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Figure  27.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  ground  truth,  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  USACE 
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Figure  28.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  AWD  method;  USACE  GPO. 


57 


556495  556500  T 

nT 

17lb  or25lb  bomb 
^  detected 

q  171b  or  251b  bomb 

not  detected 

□  range  clutter  detected 

□  range  clutter  not  detected 

+■  manual  pick 


(meters) 


USACE  GPO 

Total  Field  Magnetics 
Manual  picks 


556470 


556475 


556480 


556490 


556495 


556500 


556470 


556485  556490 


Figure  29.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth 
and  anomalies  identified  manually;  USACE  GPO. 
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4. 3. 2.2  Pueblo  of  Isleta,  NM 


The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  1 1  and  Figure 
30.  Color  coded  maps  showing  the  different  target  picking  declarations  and  ground  truth 
overlain  on  their  respective  data  are  presented  in  Figure  3 1  to  Figure  35.  The  manual,  analytic 
signal,  clustering  and  AWD  methods  all  achieved  Pd’s  greater  than  .98.  The  few  ground  truth 
targets  that  were  missed  by  these  methods  had  a  pick  just  outside  the  one  meter  tolerance  that 
was  used  to  define  a  detection.  Most  of  the  missed  targets  were  common  to  different  methods 
which  suggest  the  location  error  may  stem  from  errors  in  the  ground  truth  location  or  position 
errors  in  the  input  sensor  data  and  not  problems  with  the  target  picking  algorithms. 

The  matched  filter  method  was  again  hampered  by  the  requirement  of  the  filter  box  to  have 
complete  data  coverage.  Also,  the  matched  filter  had  the  fewest  number  of  total  picks  for  this 
area  which  points  to  a  more  conservative  threshold  chosen  by  the  analyst.  Additional  ground 
truth  could  be  detected  if  a  lower  picking  threshold  were  used.  This  is  evident  by  looking  at  the 
responses  of  the  missed  targets  (red  circles  in  Figure  33). 


Table  11 

l.  Isleta  -  Del 

tection  and 

1  location  accuracy 

Parameter  / 
Method 

Total  # 
picks 

#  matching 
ground 
truth 

Pd 

# 

<.  25m 

#  .25m 
-  .5m 

#.5- 

1.0m 

% 

within 

.5m 

Ground  Truth 

145 

Manual 

1466 

142 

0.98 

41 

67 

28 

80 

Clustering 

758 

142 

0.98 

49 

70 

23 

84 

Wavelet 

1454 

143 

0.99 

51 

72 

20 

86 

Matched  filter 

All 

131 

0.90 

51 

62 

18 

86 

Analytic  signal 

1278 

142 

0.98 

38 

78 

26 

82 

59 


Figure  30.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  Isleta. 
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Figure  3 1 .  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  analytic  signal  method;  Isleta. 
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Figure  32.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  clustering  method;  Isleta. 
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Figure  33.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  ground  truth,  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  Isleta. 
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Figure  34.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  AWD  method;  Isleta. 
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Figure  35.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth 
and  anomalies  identified  manually;  Isleta. 
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4. 3.2. 3  Seaside,  CA 

The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  12  and  Figure 
36.  Color  coded  maps  showing  the  different  target  picking  declarations  and  ground  truth 
overlain  on  their  respective  data  are  presented  in  Figure  37  to  Figure  41.  This  dataset  shows 
some  large-amplitude  anomalies  and  cluttered  areas  as  well  as  many  low  amplitude  anomalies. 
The  overall  background  noise  of  the  data  is  of  the  same  order  of  magnitude  as  the  weak 
anomalies  which  caused  problems  in  setting  the  detection  threshold.  In  order  to  pick  many  of  the 
weaker  anomalies  a  low  threshold  was  needed  but  the  low  threshold  in  turn  would  pick  many 
false  alarms  caused  by  noise  in  the  data.  The  thresholds  and  parameters  for  the  manual,  analytic 
signal,  clustering  and  matched  filter  methods  were  set  to  reduce  the  number  of  false  alanns.  On 
the  other  hand,  the  AWD  method  used  parameters  that  were  more  aggressive  which  produced  a 
higher  Pd  but  with  the  drawback  of  obtaining  multiple  alanns  for  the  same  target.  Many  of  the 
AWD  false  alarms  were  in  the  cluttered  area  located  in  the  north  central  part  of  the  map. 

Table  13  outlines  the  weights  of  the  objects  missed  by  each  of  the  methods.  For  comparison 
purposes,  a  20mm  projectile  has  a  weight  of  approximately  0.25  pounds.  Therefore,  over  90%  of 
the  missed  targets  for  each  of  the  methods  were  smaller  than  a  20mm.  This  implies  the 
thresholds  of  the  automatic  target  pickers  were  set  at  reasonable  values  to  detect  small  ordnance 
but  not  so  low  as  to  detect  very  small  pieces  of  clutter.  Most  of  the  heavy  items  (>  1  lb)  that  were 
missed  were  accompanied  by  multiple  nearby  anomalies  that  were  picked.  When  we  looked  at 
these  situations  we  noticed  that,  according  to  the  ground  truth,  the  objects  surrounding  the 
missed  targets  were  very  often  smaller  than  the  detected  ones.  An  example  is  shown  in  Figure 
40.  At  the  center  of  the  map  there  is  a  large  circle  that  encompasses  three  ground  truthed  objects. 
According  to  the  ground  truth,  the  only  object  detected  by  the  target  pickers  was  the  one  in  the 
middle  (black  circle)  and  weighed  0. libs.  The  objects  above  and  below  (red  symbols)  were  not 
detected  and  weighed  4.01bs  and  0. 1  lbs,  respectively.  The  more  likely  scenario  has  the  4.01b 
object  located  in  the  middle  and  responsible  for  the  high  amplitude  anomaly  seen  in  the  data. 

The  two  0.11b  objects  were  likely  masked  by  the  larger  anomaly  and  thus  not  picked  by  the 
automatic  target  picking  methods. 

The  combination  of  large  and  weak  anomalies  and  cluttered  areas  was  especially  difficult  for  the 
matched  filter  algorithm  and  attributed  to  the  very  low  Pd.  The  3.75m  filter  box  needed  to  detect 
the  large  anomalies  caused  problems  detecting  the  closely  spaced  weak  anomalies.  The  filter 
box  routinely  contained  several  dipoles  which  resulted  in  a  low  (MFoutput/Chisq)  value  and  thus 
a  missed  target. 


Table  12,  Seaside  -  Detection  and  location  accuracy 


Parameter  / 
Method 

Total  # 
picks 

#  matching 
ground 
truth 

Pd 

# 

<,25/i« 

#  ,25/n 
-  ,5/n 

#.5- 

1.0m 

% 

within 

.5m 

Ground  Truth 

412 

Manual 

205 

134 

0.33 

60 

53 

21 

84 
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Parameter  / 
Method 

Total  # 
picks 

#  matching 
ground 
truth 

Pd 

# 

<.2  5m 

#  .25m 
-  .5m 

#.5- 

1.0m 

% 

within 

.5m 

Clustering 

160 

102 

0.25 

24 

46 

32 

69 

Wavelet 

616 

246 

0.60 

114 

100 

32 

87 

Matched  filter 

123 

62 

0.15 

24 

29 

9 

85 

Analytic  signal 

346 

220 

0.53 

73 

109 

38 

83 

Figure  36.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  Seaside. 


Table  13.  Seaside  -  Weight  of  targets  missed 


Parameter  / 
Method 

#  <0.1  lbs 

#0.1lbs  -0.25lbs 

#  >0.25lbs 

%  <0 .25lbs 

Manual 

217 

44 

17 

94 

Clustering 

235 

52 

23 

93 

Wavelet 

134 

22 

7 

96 

Matched  filter 

260 

58 

32 

91 

Analytic  signal 

154 

24 

13 

93 
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Figure  37.  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  analytic  signal  method;  Seaside. 
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Figure  38.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  clustering  method;  Seaside. 
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Figure  39.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  ground  truth,  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  Seaside. 
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Figure  40.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  AWD  method;  Seaside.  The  larger  circle  in  the 
center  of  the  figure  is  an  example  of  an  object  with  questionable  ground  truth. 
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Figure  41.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth 
and  anomalies  identified  manually;  Seaside. 
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4. 3. 2. 4  Jefferson  Proving  Ground,  IN 


The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  14  and  Figure 
42.  Color  coded  maps  showing  the  different  target  picking  declarations  and  ground  truth 
overlain  on  their  respective  data  are  presented  in  Figure  43  to  Figure  47.  The  manual,  analytic 
signal,  clustering  and  AWD  methods  all  performed  similarly  and  achieved  Pd’s  greater  than  .92. 
Similar  targets  were  missed  by  the  different  methods  with  two  common  themes.  First,  the 
majority  of  the  missed  objects  were  masked  by  the  responses  of  nearby  objects.  On  several 
occasions  two  or  more  objects  were  located  within  a  few  meters  of  each  other.  The  AWD 
method  appeared  to  do  a  better  job  than  the  other  methods  with  these  overlapping  situations. 
Second,  a  few  objects  had  weak  responses  that  were  at  or  just  below  the  detection  thresholds 
used.  The  data  contained  a  low  amplitude  feature  in  the  north-south  direction  that  cut  across  the 
western  portion  of  the  area.  The  threshold  for  the  automatic  detectors  needed  to  be  set  at  a  level 
just  above  the  response  of  this  feature  to  minimize  the  number  of  false  alarms  caused  by  the 
feature.  This  resulted  in  a  slightly  higher  than  desired  threshold  which  produced  a  few  missed 
objects. 

These  data  were  characterized  by  positioning  problems  as  seen  by  the  herringbone  nature  of  the 
anomalies  shown  in  Figure  48.  The  data  problems  caused  multiple  AWD  declarations  over 
several  of  the  objects.  These  data  position  problems  in  conjunction  with  the  closely  spaced 
anomalies  caused  problems  for  the  matched  filter  method.  In  contrast  to  the  AWD,  the  match 
filter  resulted  in  fewer  detection  because  the  data  had  a  poorer  match  to  the  dipole  model.  This 
resulted  in  a  smaller  MF  output  as  well  as  a  larger  Chisq  so  the  (MFoutput/Chisq)  ratio  used  was 
significantly  reduced. 


Table  14.  JPG  -  Detection  and  location  accuracy 


Parameter  / 
Method 

Total  # 
picks 

#  matching 
ground 
truth 

Pd 

# 

<.  25m 

#  .25m 
-  .5m 

#.5- 

1.0m 

% 

within 

.5m 

Ground  Truth 

190 

Manual 

532 

180 

0.95 

67 

84 

29 

84 

Clustering 

359 

174 

0.92 

80 

70 

24 

86 

Wavelet 

491 

180 

0.95 

85 

76 

19 

89 

Matched  filter 

309 

137 

0.72 

73 

46 

18 

87 

Analytic  signal 

393 

174 

0.92 

61 

82 

31 

82 

73 


Figure  42.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  JPG. 
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Figure  43.  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  analytic  signal  method;  JPG. 
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Figure  44.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  clustering  method;  JPG. 
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Figure  45.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  ground  truth,  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  JPG. 
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Figure  46.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth, 
manual  picks  and  anomalies  identified  by  the  AWD  method;  JPG. 
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Figure  47.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  ground  truth 
and  anomalies  identified  manually;  JPG. 
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Figure  48.  Total  field  magnetic  data  over  a  portion  of  the  eastern  edge  of  the  JPG  survey  area. 
The  X's  show  the  multiple  wavelet  picks  caused  by  positioning  problems.  The  color  scale  is  the 
same  as  in  Figure  46. 


80 


4. 3.2. 5  Aberdeen  Proving  Ground,  MD 

As  previously  mentioned,  a  limited  number  of  ground  truth  information  has  been  released  over 
the  open  field  area  at  APG.  Therefore  our  target  dig  sheets  were  scored  against  the  emplaced 
targets  by  analysts  from  the  Institute  for  Defense  Analyses  (IDA).  The  accurate  ground  truth 
information  at  APG  enabled  the  calculation  of  Pd  and  rBAR  as  well  as  statistics  of  the  location 
accuracy  of  the  target  declarations  for  each  of  the  picking  methods.  The  perfonnance  of  the 
different  target  picking  methods  is  summarized  in  Table  15  and  Figure  49  and  Figure  50.  The 
numerator  for  the  Pd  represents  the  total  number  of  ferrous  ground  truth  items  with  no  nearest 
neighbor  within  2m  (2m  flag),  AND  (for  ordnance)  not  deeper  than  the  1  lx  depth,  AND  with  an 
alarm  ("suspected  target")  within  the  0.5m  scoring  halo  (radius).  No  clutter  was  removed  by  the 
diameter  to  depth  filter.  For  large  targets  (e.g.,  155mm  shell),  the  halo  was  elongated  into  an 
ellipse  along  the  long  axis  of  the  shell.  The  semi-major  axis  is  half  the  item's  length  projected 
onto  the  ground  plane,  plus  0.5m.  The  denominator  for  the  Pd  calculation  is  the  total  number  of 
true  clutter  and  shallower  than  1  lx  ferrous  ordnance  with  no  nearest  neighbor  within  2m.  All  the 
calculations  were  rounded  to  5%  to  protect  the  truth.  The  ROC  curves  in  Figure  49  show  that  the 
manual  and  analytic  signal  methods  had  the  highest  Pd  with  the  AWD  method  close  behind. 
These  three  methods  also  had  the  highest  number  of  background  alarms  which  indicates  that 
parameters  were  set  to  pick  targets  close  to  the  noise  background  of  the  data.  The  last  5%  of 
detections  came  at  the  expense  of  more  than  doubling  the  number  of  background  alarms.  The 
location  accuracy  for  the  different  picking  methods  was  fairly  similar  with  the  matched  filter 
method  producing  the  most  accurate  results.  The  matched  filter  method  convolves  the  data  with 
a  magnetic  dipole  signal  model  which  outputs  target  parameters.  This  would  produce  more 
accurate  locations  than  the  other  methods  that  only  use  the  locations  and  amplitudes  of  positive 
and  negative  peaks  in  the  magnetic  data  to  find  the  target  location. 

The  following  comments  on  the  results  were  received  from  IDA. 

•  Large  anomalies  caused  shadowing  of  one  target  over  another  from  2-6m  separation. 
Ordnance  up  to  105mm  projectiles  were  missed  because  of  these  overlapping  signals.  In 
some  cases,  there  was  no  suspected  target  (alarm),  in  others  there  was  a  near  miss  alarm 
(within  a  lm  halo  however).  Note  that  the  lm  halo  would  only  hit  more  truth  targets  as 
onsies,  twosies.  Near  miss  was  a  failure  mode,  but  not  the  only  one. 

•  The  matched  filter  had  trouble  with  the  bomblets  and  submuntions. 

•  Some  targets  were  only  found  by  the  manual  mode;  this  mode  did  very  well.  Several 
shadowed  targets  were  found  with  manual. 

•  All  had  some  trouble  with  20mm’ s,  manual  did  the  best. 

We  also  selected  80  anomalies  at  APG  to  submit  to  the  magnetic  inversion  algorithm  found  in 
UX-Analyze  (UXO  analysis  software  packaged  developed  under  ESTCP  project  MM-0210)  in 
order  to  test  the  ability  of  each  method  to  extract  the  anomalies’  spatial  extent.  Of  the  80 
anomalies  approximately  80%  matched  ferrous  ground  truth  within  a  0.5m  halo  with  no  nearest 
neighbor  within  2m.  Table  16,  Figure  5 1  and  Figure  52  summarizes  the  location,  depth  and  size 
accuracies  for  the  targets  that  matched  a  ground  truth  item.  Overall,  all  the  methods  produced 
similar  results  with  the  location  and  depth  accuracies  being  better  than  the  size  estimates.  As 
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expected  the  locations  output  from  the  inversion  algorithms  were  more  accurate  than  those 
output  directly  from  the  picking  methods  (Figure  51).  Both  location  and  depth  estimates  were 
sufficiently  accurate  to  allow  dig  teams  to  easily  find  the  object  of  interest.  Color  coded  maps 
showing  the  different  target  picking  declarations  and  locations  of  the  80  anomalies  selected  for 
inversion  are  presented  in  Figure  53  to  Figure  56. 


Table  15.  APG  -  Detection  ant 

location  accuracy 

Parameter  / 
Method 

Total  # 
picks 

Pd 

rBAR 

Average  miss 
Distance  (m) 

Manual 

707 

0.9 

0.58 

.204 

Clustering 

444 

0.8 

0.335 

.196 

Wavelet 

743 

0.85 

0.615 

.212 

Matched  filter 

206 

0.7 

0.105 

.171 

Analytic  signal 

620 

0.9 

0.495 

.225 

ROCs  (Detection  picks) 


rBAR 


- AWD 

Analytic  signal 
Clustering 
Matched  filter 
- Manual 


Figure  49.  Receiver  operating  curve  showing  the  number  of  ground  truth  objects  detected  by 
each  method  versus  Background  alarm  rate;  APG. 
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Figure  50.  Graph  showing  the  average  miss  for  Easting  and  Northing  (in  meters).  The  range  bars 
are  symmetrical  standard  deviations. 
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Table  16  APG  -  Location  and  size  accuracy  of  the  80  targets  selected  for  inversion. 


Method  / 

Parameter 

Analytic 

signal 

Clustering 

Manual 

Matched 

filter 

AWD 

%  of  UXA  targets  matched  within  a  0.5m 
halo  (2m  flag)  of  the  ground  truth  location 
using  the  initial  XY 

79% 

77% 

81% 

81% 

81% 

%  of  UXA  targets  matched  to  a  truth  item 
(2m  flag)  also  within  ,3m  radius  using  the 
fitted  XY 

93% 

92% 

92% 

90% 

92% 

Average  miss  distance  of  initial  picks  (m) 

.217 

.177 

.193 

.156 

.196 

Average  miss  distance  of  fitted  picks  (m) 

.133 

.134 

.134 

.139 

.135 

%  of  matched  UXA  targets  within  .3m  of  the 
ground  truth  depth. 

97% 

94% 

97% 

97% 

94% 

Average  miss  depth  (m) 

-.074 

-.057 

-.067 

-.072 

-.082 

Standard  deviation  of  depth  misses  (m) 

.157 

.181 

.148 

.151 

.162 

%  of  matched  UXA  targets  (only  ordnance, 

2m  flag,  11x  flag)  with  a  size  factor  less 
than  or  equal  to  0.3.  Size  factor  = 
ABS(true_size  -  estimatedsize)  /  true  size. 
Does  not  include  5001b  Bombs. 

47% 

39% 

44% 

44% 

44% 
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Figure  5 1 .  Comparison  of  the  location  accuracy  of  the  positions  output  directly  from  the  different 
picking  methods  (left)  and  the  locations  output  after  passing  the  selected  data  to  the  magnetic 
fitting  algorithm  in  UX -Analyze  (right). 


Figure  52.  Depth  fit  accuracy  of  the  different  picking  methods.  Data  extracted  from  each  picking 
method  were  passed  to  a  magnetic  inversion  algorithm  to  estimate  the  target's  depth. 


84 


402760  402780 


402680 


402700 


402720 


402740 


©£j 

fc®  ® 

jp 

^<1 

MI 

i 

jgjP 

s 

If 

g|jg§ 

if 

I|||j|| 

M 

sp 

ll 

I 

||^ 

tjy 

pip 

1 

y*p®’Q 

f§j 

d 

Hfe 

i§£  <s 

402680  402700  402720  402740  402760  402780 


146.2 

142.3 

138.5 

134.6 
130  8 

1269 

123.1 

119.2 

115.4 

111.5 

107.7 

103.8 

1000 

962 

92  3 

88.5 

84.6 

80.8 

769 

73.1 

69.2 
65.4 

5 

61.5 

57.7 

538 

50.0 

46.2 

42.3 

38  5 

34  6 

— 

30.8 

— 

26.9 

— 

23.1 

— 

192 

— 

15.4 

— 

11.5 

— 

7.7 

3.8 

nT/m 

10  0  10  20 


(meters) 

NAD83/  UTM  zone  18N 


APG  -  Portion  of  Open  field  grid 

Analytic  Signal 
Analytic  signal  picks 


O  manual  pick  (red  symbols 
represent  targets  selected 
for  characterization) 

+  analytic  signal  pick 


Figure  53.  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  manual  picks  and 
anomalies  identified  by  the  analytic  signal  method;  APG. 
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Figure  54.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  manual  picks 
and  anomalies  identified  by  the  clustering  method;  APG. 
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Figure  55.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  APG. 
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Figure  56.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  manual  picks 
and  anomalies  identified  by  the  AWD  method;  APG. 
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4. 3.2. 6  WAA  Airborne  survey  -  Pueblo  Precision  Bombing  Range  #2,  CO 

The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  17  ,  Figure  57 
and  Figure  58.  Color  coded  maps  showing  the  different  target  picking  declarations  and  manual 
picks  overlain  on  their  respective  data  are  presented  in  Figure  59  to  Figure  62.  The  Pueblo 
Airborne  survey  did  not  have  any  ground  truth  so  the  automatic  target  picks  were  compared  to 
the  manual  picks.  The  different  methods  had  similarly  poor  Pd’s  but  a  range  of  Pfa’s.  The  main 
reason  for  the  poor  Pd’s  was  the  high  background  geology  throughout  the  area.  Figure  58  shows 
that  all  the  methods  stopped  picking  targets  at  roughly  the  same  threshold.  The  parameters  and 
thresholds  for  each  of  the  methods  were  set  to  maximize  the  number  of  targets  selected  while 
keeping  the  number  of  picks  related  to  geology  at  a  minimum. 

Figure  63  shows  noise  levels  at  various  locations  over  the  survey  area.  The  noise  was  estimated 
by  calculating  the  standard  deviation  of  the  data  contained  within  each  box  displayed  on  the  map. 
The  boxes  were  located  in  areas  that  contained  various  noise  levels  and  were  void  of  any  manual 
picks.  The  noise  ranged  from  0. 16nT  to  2.28nT  which  is  a  wide  range  for  these  data.  The 
manual  method  had  the  advantage  of  dynamically  altering  the  picking  threshold  to  the 
surrounding  noise  levels  and  thus  was  able  to  select  targets  in  the  southern  portion  of  the  area 
where  the  noise  levels  were  lower.  In  the  southern  area,  the  automatic  pickers  selected  few 
targets  because  their  thresholds  could  not  be  set  low  enough  to  detect  these  targets  without 
adding  thousands  of  additional  picks  due  to  geology.  Conversely,  in  the  northern  portion  of  the 
survey  area  where  the  geologic  noise  is  much  greater,  many  of  the  automatic  pickers  selected 
numerous  anomalies  that  were  geology  related  as  shown  in  Figure  64.  This  effect  is  especially 
seen  in  the  wavelet  and  analytic  signal  methods  and  resulted  in  their  higher  Pfa’s.  On  the  other 
hand,  the  (MFoutput/Chisq)  did  a  good  job  at  reducing  the  effect  of  the  geology. 

Even  though  the  manual  method  performed  well  with  these  data  it  missed  a  few  obvious  strong 
anomalies  that  all  the  automatic  methods  found  as  shown  in  Figure  65.  Here  we  see  some 
anomalies  shown  by  the  black  squares  that  were  missed  by  the  manual  method.  It  is  likely  these 
were  missed  because  of  the  “human  factor”.  It  took  the  manual  analyst  9  hours  to  pick  all  the 
targets  in  this  area.  This  was  a  lengthy  endeavor  and  during  the  systematic  process  of  zooming 
and  scrolling  through  the  data  this  small  section  of  data  was  inadvertently  missed. 
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Table  17.  WAA  Pueblo  Airborne  -  Detection  and  location  accuracy 


Parameter  / 
Method 

Total  # 
picks 

Pd 

Pfa 

#  within 
lm 

#  1  - 
1.5  m 

#1.5- 
2.0  m 

%  within 
1.5  m 

Manual 

383 

Clustering 

387 

0.55 

0.46 

153 

41 

17 

92 

Wavelet 

547 

0.53 

0.90 

138 

47 

17 

92 

Matched  filter 

323 

0.50 

0.34 

140 

41 

12 

94 

Analytic  signal 

397 

0.53 

0.50 

185 

17 

2 

99 

Figure  57.  Graph  showing  Pd  versus  the  number  of  false  alarms  (automatic  picks  not  matching  a 
manual  picks);  Pueblo  airborne. 


90 


WAA  Airborne  Pueblo  -  Manual  picks  detected 


400 


350 


300 


250 

■O 

Q) 

&  200 

d) 

TJ 

% 

150 


100 


50 

0  V  , 

0  50  100  150  200  250  300  350  400 

#  of  Manual  targets  sorted  by  decreasing  amplitude 


Figure  58.  Graph  showing  the  number  of  manual  picks  detected  by  each  method  as  a  function  of 
signal  amplitude;  Pueblo  airborne. 
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Figure  59.  Color-coded  map  showing  the  analytic  signal  data  overlain  by  the  manual  picks  and 
anomalies  identified  by  the  analytic  signal  method;  Pueblo  airborne. 
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Figure  60.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  manual  picks 
and  anomalies  identified  by  the  clustering  method;  Pueblo  airborne. 
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Figure  61.  Color-coded  map  showing  the  (Matched  fdter  output  /  Chi  square)  data  overlain  by 
the  manual  picks  and  anomalies  identified  by  the  matched  filter  method;  Pueblo  airborne. 
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Figure  62.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  manual  picks 
and  anomalies  identified  by  the  AWD  method;  Pueblo  airborne. 
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Figure  63.  Color  coded  map  of  the  total  field  magnetic  showing  the  noise  at  various  locations. 
The  noise  was  estimated  by  calculating  the  standard  deviation  of  the  data  contained  within  each 
box. 
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Figure  64.  Color  coded  map  of  the  total  field  magnetic  data  showing  the  numerous  picks  due  to 
geology  by  the  different  target  pickers  in  the  northern  portion  of  the  Pueblo  airborne  survey  area. 
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Figure  65.  Color  coded  map  of  the  total  field  magnetic  data  showing  some  targets  missed  by  the 
manual  picker  but  detected  by  all  the  automatic  methods. 
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4. 3.2. 7  WAA  Transect  survey  -  Pueblo  Precision  Bombing  Range  #2,  CO 

The  performance  of  the  different  target  picking  methods  is  summarized  in  Table  18,  Figure  66 
and  Figure  67.  Color  coded  maps  showing  a  section  of  the  different  target  picking  declarations 
and  manual  picks  overlain  on  their  respective  data  are  presented  in  Figure  68.  The  analyzed 
transects  only  covered  two  anomalies  with  known  ground  truth;  therefore  the  automatic  target 
picks  were  compared  to  the  manual  picks.  As  stated  earlier,  the  matched  filter  could  not  be  run 
on  this  data  set  because  the  filter  box  needed  would  be  larger  than  the  1.75m  width  of  the 
transect  data.  The  clustering  and  analytic  signal  methods  produced  similar  results  for  these  data. 
Both  methods  picked  the  larger  targets  with  few  false  alarms  when  compared  to  the  manual 
declarations.  As  Figure  67  shows  the  thresholds  for  the  two  methods  were  set  higher  than  the 
wavelet  method  and  the  manual  method.  Because  the  object  of  these  data  was  WAA  the 
thresholds  were  set  at  reasonable  values. 

The  wavelet  method  had  a  higher  Pd  but  at  the  cost  of  many  more  false  alarms.  136  of  these 
false  alarms  are  picks  that  occurred  outside  the  extents  of  the  data.  A  few  examples  can  be  seen 
in  Figure  68.  They  were  caused  by  extrapolation  of  the  data  when  creating  the  input  grid  to  the 
wavelet  algorithm.  Most  of  the  remaining  false  alarms  were  low  amplitude  declarations  that  did 
not  match  a  manual  pick  as  shown  in  Figure  69.  These  additional  picks  (black  crosses)  as  well 
as  many  of  the  manual  picks  (black  circles)  that  were  not  selected  by  the  automatic  methods 
were  at  amplitudes  that  were  very  close  to  the  background  noise  of  the  data.  Without  complete 
ground  truth,  we  cannot  determine  if  these  were  valid  picks  due  to  a  metallic  object  or  caused  by 
noise. 


Table  18.  WAA  Pueblo  Transects  -  Detection  and  location  accurac 


Parameter  / 
Method 

Total  # 
picks 

Pd 

Pfa 

#  within 
lm 

#1- 
1.5  m 

#  1.5- 
2.0m 

%  within 
1.5  m 

Manual 

887 

Clustering 

454 

0.49 

0.02 

375 

40 

24 

95 

Wavelet 

1049 

0.76 

0.42 

506 

117 

52 

92 

Analytic  signal 

505 

0.55 

0.02 

474 

9 

1 

100 

98 


Figure  66.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  Pueblo  transects. 


Figure  67.  Graph  showing  the  number  of  ground  truth  objects  detected  by  each  method  as  a 
function  of  signal  amplitude;  Pueblo  transects. 
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Figure  68.  Color-coded  map  showing  the  total  field  magnetic  data  overlain  by  the  manual  picks 
and  anomalies  identified  by  the  clustering  method;  Pueblo  transects. 
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Figure  69.  Section  of  transect  magnetic  data  showing  some  low  amplitude  declaration  by  the 
different  picking  methods.  The  analytic  signal,  clustering,  wavelet  and  manual  picks  are  plotted 
as  black  plus  sign,  red  box,  black  cross  and  black  circle,  respectively. 
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4.3.3  Target  Picking  Time 

The  actual  time  it  took  to  set  up  the  optimum  parameters  and  run  the  automatic  target  pickers 
were  recorded  for  each  algorithm  and  data  set.  We  also  measured  the  setup  time  and  processing 
time  separately  because  the  setup  time  will  be  major  component  of  the  total  time  for  the 
relatively  small  data  sets  we  used  in  this  demonstration.  However,  on  large  data  sets  that  have 
relatively  consistent  data  characteristics  the  setup  time  will  be  negligible  when  compared  to  the 
overall  time  to  pick  all  the  targets.  The  total  picking  time  for  the  different  methods  is  shown  in 
Figure  70.  The  analytic  signal  was  by  far  the  quickest  of  the  methods  followed  by  the  AWD. 
The  matched  filter  and  clustering  algorithms  gave  similar  times  with  the  manual  picking  being 
the  slowest  as  expected.  It  was  noted  that  for  very  small  data  sets  with  few  anomalies,  such  as 
the  USAGE  GPO,  the  manual  picking  performed  quicker  than  the  clustering  and  matched  filter 
methods.  This  was  due  entirely  to  the  setup  time.  Figure  71  presents  the  results  if  we  remove 
the  setup  time  and  only  compare  the  processing  times.  This  graph  clearly  shows  the  automatic 
picking  methods  are  much  quicker  (4-50  times  faster)  than  the  manual  method  in  all  cases.  We 
note  that  for  the  transect  data  the  speed  of  the  AWD  algorithm  is  greatly  affected.  This  is 
because  of  the  large  spatial  extent  of  the  transects  and  the  requirement  that  the  data  be  gridded  at 
a  small  (relative  to  the  spatial  extent  of  the  data)  grid  cell  size.  On  the  other  hand  the  clustering 
algorithm  works  on  the  actual  profile  data  and  not  the  gridded  data  so  is  not  affected  by  spatial 
extent  of  the  transects. 
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Figure  70.  Graph  showing  the  total  time  to  analyze  the  data  sets  for  the  different  target  picking 
methods. 
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Target  picking  processing  time 
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Figure  7 1 .  Graph  showing  the  processing  time  to  analyze  the  data  sets  for  each  of  the  target 
picking  methods. 


4.3.4  Qualitative  Metrics 

All  the  target  picking  methods  were  incorporated  into  the  Oasis  montaj™  platfonn.  This  made  it 
easy  for  users  experienced  in  data  analysis  and  Oasis  montaj™  to  execute  the  software.  The 
analysts  were  also  able  to  directly  access  the  graphical  presentations  and  appropriate  tools  in 
Oasis  montaj™  to  assist  in  their  analysis.  The  input  dialogs  and  basic  procedures  to  run  the 
software  are  straight  forward  for  all  the  algorithms  but  an  experienced  user  would  be  required  to 
determine  which  target  picking  method  to  use  and  what  how  to  set  the  parameters.  Currently,  the 
software  has  limited  help  for  the  different  algorithms  and  no  user  manual  to  direct  the  user  on  the 
advantages  and  disadvantages  of  the  different  algorithms  and  how  to  set  the  parameters  for  each 
of  the  algorithms.  There  are  few  parameters  to  set  for  the  analytic  signal  and  matched  filter 
algorithms  and  they  are  fairly  simple  to  set.  Conversely,  the  AWD  and  clustering  methods  have 
many  parameters  that  require  setting  to  achieve  the  best  results. 

During  this  demonstration,  the  clustering  algorithm  encounter  a  few  memory  related  bugs  which 
have  been  corrected.  As  of  November  2007,  the  current  Oasis  montaj™  release  is  6.4  and  we  do 
not  experience  any  systematic  difficulties  or  intermittent  bugs  while  using  the  analytic  signal, 
clustering  or  matched  filter  algorithms.  The  AWD  algorithm  is  also  operational  but  still  needs  to 
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be  run  using  Oasis  montaj™  V6.2  because  of  incompatibilities  between  the  .NET  versions  used 
in  SkyNet  and  Oasis  montaj.  This  issue  is  expected  to  be  resolved  in  the  near  future  as  SkyNet  is 
upgraded  to  .NET  version  2.0. 


4.3.5  Discussion 

Overall,  the  manual  method  proved  to  be  the  best  at  picking  valid  targets  especially  in  areas  with 
varying  amounts  of  geology  and  background  noise.  In  general,  the  manual  picker  was  able  to  set 
a  lower  threshold  for  each  data  set  than  the  automatic  methods  because  he  can  screen  out  the 
anomalies  that  are  caused  by  noisy  data  or  geology  and  pick  the  small  amplitude  anomalies 
located  in  the  quieter  areas  as  seen  in  the  Pueblo  airborne  data  set.  This  ability  is  also  useful 
when  dealing  with  poor  quality  data  seen  in  the  JPG  data.  The  manual  picker  was  able  to 
interpret  positioning  problems  that  caused  some  large  anomalies  to  appear  to  be  several  small 
anomalies  and  only  make  one  declaration  whereas  some  of  the  automatic  picks  had  multiple 
declarations.  The  main  drawbacks  to  the  manual  method  are  time,  operator  bias  and  operator 
error.  In  this  demonstration  the  manual  method  was  4  to  50  times  slower  (not  including  setup 
time)  than  the  automatic  methods  depending  on  the  data  set.  We  also  saw  in  the  Pueblo  airborne 
data  how  operator  fatigue  or  error  resulted  in  some  missed  targets.  Although  only  one  manual 
analyst  was  used  for  this  demonstration  we  see  signs  of  operator  bias.  The  manual  picking 
threshold  for  all  the  data  sets  was  very  close  to  the  background  noise  level  except  for  the  Seaside 
data  which  found  significantly  fewer  matches  to  ground  truth  objects  than  the  analytic  signal  and 
wavelet  methods.  This  bias  is  likely  to  be  even  more  pronounced  if  different  analysts  are  used. 

Of  the  automatic  methods  the  analytic  signal  and  wavelet  method  gave  the  best  results  overall 
but  each  of  the  methods  had  their  own  strengths  and  weaknesses  and  the  best  method  to  use  was 
very  data  dependent.  A  general  observation  for  all  the  automatic  pickers  is  that  they  should  not 
be  run  blindly.  The  analyst  should  carefully  choose  their  parameters  and  analyze  the  results. 

The  process  of  iteratively  changing  parameters  and  visual  review  of  the  results  was  essential  in 
selecting  the  best  parameters. 

The  analytic  signal  method  was  the  fastest  and  simplest  method.  It  worked  well  on  a  range  of 
target  sizes  and  densities.  It  was  able  to  counteract  some  of  the  over  picking  problems  due  to 
data  quality  issues  by  additional  grid  filtering  passes.  Its  main  weakness  was  picking  targets  in 
areas  with  a  variety  of  background  geology  or  noise  levels.  Because  a  hard  threshold  is  set,  it 
can  either  be  tuned  to  select  the  weak  anomalies  and  pick  numerous  false  alarms  or  set  higher 
with  the  consequence  of  missing  the  weak  anomalies  but  minimizing  false  alarms. 

The  wavelet  method  worked  well  on  a  range  of  target  sizes  and  densities.  It  performed  better 
than  the  other  methods  at  picking  targets  that  were  clustered.  It  was  fairly  quick  to  run  on  total 
coverage  surveys  but  much  less  efficient  when  run  on  transect  data.  Its  main  weaknesses  were 
the  complexity  of  the  parameters  and  over  picking  in  areas  with  geology,  noise  or  poor  data 
quality.  The  default  set  of  parameters  provided  an  adequate  starting  point  and  improvements 
were  easily  gained  by  adjusting  the  amplitude  thresholds.  But  further  improvements  in  the 
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detection  rate  required  expertise  or  at  least  advanced  familiarity  with  the  AWD  and  the 
sensitivity  of  its  parameters. 

The  clustering  method  performed  well  detecting  isolated  targets  with  similar  sizes.  When  a 
range  of  target  sizes  were  encountered  and  the  parameters  were  set  to  detect  the  large  anomalies 
any  small  anomaly  close  to  the  large  anomaly  tended  to  be  missed.  The  clustering  algorithm  also 
had  problems  picking  anomalies  in  areas  with  high  geology  and  noise.  Overall,  it  ran  slower 
than  the  analytic  signal  on  total  survey  areas  but  was  more  efficient  on  transect  data  because  it 
works  on  the  profile  data  and  not  a  grid.  It  also  took  longer  to  setup  because  there  were  multiple 
parameters  that  need  to  be  tuned. 

The  matched  filter  method  performed  the  worst.  It  was  hindered  by  its  requirement  to  have  total 
data  coverage  within  the  filter  box.  This  would  increase  geophysical  data  collection  costs 
because  additional  data  would  need  to  be  collected  around  the  perimeter  of  the  survey  area.  It 
ran  the  slowest  of  the  automatic  methods  but  the  limited  number  of  parameters  helped  reduce  its 
complexity.  It  had  difficulty  with  overlapping  targets,  range  of  target  sizes  and  poor  data  quality. 
The  one  area  it  excelled  at  was  picking  targets  in  areas  with  a  variety  of  geology. 

Implementation  of  the  automatic  picking  methods  used  in  this  demonstration  should 
considerably  reduce  the  time  and  thus  cost  required  to  pick  anomalies  when  compared  to  the 
manual  method.  The  amount  of  cost  savings  will  depend  on  the  data.  In  areas  with  isolated 
anomalies  and  low  background  noise  or  geology  the  cost  savings  will  be  maximized  because  the 
automatic  pickers  are  able  to  detect  over  90%  of  the  anomalies  in  a  fraction  of  the  time  compared 
to  the  manual  method.  As  the  geologic  noise  increases  or  data  quality  decreases  the  cost  savings 
will  diminish  but  still  should  be  significant.  Even  the  toughest  data  sets  in  this  demonstration 
will  produce  cost  savings.  For  example,  the  airborne  WAA  data  set  at  Pueblo  contained  an 
abundance  of  geology  that  resulted  in  Pd’s  of  around  0.5  for  the  automatic  pickers.  The  analytic 
signal  required  less  than  an  hour  to  setup  and  run  while  the  manual  method  took  9  hours.  Even  if 
it  takes  an  additional  5  hours  of  manual  picking  to  select  the  other  half  of  the  targets,  the  time 
savings  will  be  three  hours  or  33%. 

In  conclusion,  the  best  use  of  the  automatic  target  pickers  may  be  to  quickly  and  consistently 
select  all  the  strong  isolated  targets  by  setting  the  picking  parameters  to  minimize  false  alarms. 
The  remaining  anomalies  can  then  be  selected  using  the  manual  method.  This  combines  the 
strengths  of  each  method  and  will  result  in  a  better  product  than  using  only  the  automatic 
methods  and  cost  savings  compared  to  using  only  the  manual  method. 
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5.  Cost  Assessment 


5.1  Cost  Reporting 

This  demonstration  focuses  on  the  detection  of  anomalies  observed  in  magnetic  data.  As  such,  it 
encompasses  only  a  small  subset  of  costs  that  are  typically  associated  with  full-scale 
demonstrations.  The  relevant  cost  categories  and  actual  costs  for  this  demonstration  are  shown 
in  Table  19.  Time  spent  during  data  analysis  was  logged  as  described  in  Section  4.2. 


Table  19.  Cost  categories  and  details 


Cost  Category 

Details 

Sub  Category 

Time 

(hours) 

Costs* 

($) 

Data  Analysis 

Setup  parameters  for 
identifying  anomalies 

Analytic  signal 

1.5 

150 

Wavelet 

8.2 

820 

Clustering 

23.7 

2370 

Matched  filter** 

21.9 

2190 

Manual 

0 

0 

Data  Analysis 

Running  automatic  target 
pickers 

Analytic  signal 

1.7 

170 

Wavelet 

5.4 

540 

Clustering 

3.8 

380 

Matched  filter** 

6.2 

620 

Manual 

43 

4300 

Reporting 

Technical  Report 

— 

320 

32,000 

TOTAL 

43,540 

*  assumes  an  fully-loaded  labor  rate  of  $  100  per  hour 

**  Matched  filter  hours  only  includes  6  of  the  7  data  sets  because  it  could  not  be  run  on  the 
WAA  Pueblo  transect  data 


5.2  Cost  Analysis 

The  baseline  alternative  to  the  automatic  target  pickers  is  manual  target  selection.  Implementing 
the  automatic  picking  methods  used  in  this  demonstration  should  considerably  reduce  the  time 
and  thus  cost  required  to  pick  anomalies  when  compared  to  the  manual  method.  The  amount  of 
cost  savings  will  depend  on  the  data  which  will  determine  the  most  efficient  automatic  method. 
In  areas  with  isolated  anomalies  and  low  background  noise  or  geology  the  cost  savings  will  be 
maximized  because  the  automatic  pickers  are  able  to  detect  over  90%  of  the  anomalies  in  a 
fraction  of  the  time  compared  to  the  manual  method.  As  the  geologic  noise  increases  or  data 
quality  decreases  the  cost  savings  will  diminish  but  still  should  be  significant.  Even  the  toughest 
data  sets  in  this  demonstration  will  produce  cost  savings.  For  example,  the  airborne  WAA  data 
set  at  Pueblo  contained  an  abundance  of  geology  that  resulted  in  Pd’s  of  around  0.5  for  the 
automatic  pickers.  The  analytic  signal  required  less  than  an  hour  to  setup  and  run  while  the 
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manual  method  took  9  hours.  Even  if  it  takes  an  additional  5  hours  of  manual  picking  to  select 
the  other  half  of  the  targets,  the  time  savings  will  be  three  hours  or  33%. 

The  parameter  setup  time  represented  a  large  portion  of  the  processing  time  for  the  relatively 
small  data  sets  used  in  this  demonstration.  The  ground  based  surveys  ranged  from  0.3  to  10 
acres  in  size.  On  the  other  hand,  large  scale  cleanup  efforts  are  typically  hundreds  of  acres  so  the 
setup  costs  will  be  a  trivial  portion  of  the  total  costs.  Assuming  the  geology,  data  quality  and 
data  objectives  for  the  survey  area  are  relatively  constant;  the  automatic  target  picking 
parameters  should  need  minimal  changes.  If  we  compare  only  the  costs  to  actually  run  the 
automatic  methods  to  the  manual  method,  we  find  the  cost  of  the  automatic  pickers  is  roughly 
10%  of  the  cost  of  the  manual  method.  If  we  factor  in  a  very  conservative  40%  additional  time 
to  fill  in  missing  anomalies  using  the  manual  method,  there  is  still  a  cost  savings  of  50% 
achieved  by  using  the  automatic  picking  methods. 
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6.  Implementation  Issues 


6.1  Environmental  Checklist 

Not  applicable  to  this  demonstration. 


6.2  Other  Regulatory  Issues 

Not  applicable  to  this  demonstration. 


6.3  End-User  Issues 

The  end-users  of  this  data  analysis  technology  include  private  contractors  that  conduct 
geophysical  investigations  in  support  of  UXO  clean  up  programs  and  governmental  employees 
that  provide  technical  oversight.  This  demonstration  will  introduce  the  stakeholders  and  end- 
users  to  the  applicability  of  different  automatic  target  pickers  to  their  data.  This  basic 
information  will  help  to  improve  the  results  of  future  geophysical  investigations  conducted  by 
others.  The  market  for  this  type  of  guidance  document  includes  all  practicing  geophysical 
service  firms  currently  working  in  the  UXO  industry.  The  target  picking  algorithms  would 
ideally  be  transitioned  to  the  user  community  through  Geosoft  as  part  of  their  proposed  “UXO 
System”. 
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